Skip to main content
ExLibris
  • Subscribe by RSS
  • Ex Libris Knowledge Center

    Working with Normalization Rules

    Translatable
    To work with normalization rules, you must have the following role:
    • Catalog Administrator
    • Catalog Manager
    • Cataloger
    Normalization rules provide the building blocks for making enhancements to MARC 21 records. These enhancements can be made to individual records in the MD Editor (Edit > Enhance the Record) or to a group of records using sets.
    In order to process record enhancements to an individual record, use the Enhance Record option (refer to the section MD Editor Menu and Toolbar Options and Enhance the Record) or Apply Changes to an individual record when you use the Preview normalization rule capability (see the procedure To preview the outcome of a rule file).
    In order to process record enhancements to a set of records, you need to create a process using the MarcDroolNormalization or DcDroolNormalization tasks (see Working with Normalization Processes) and specify the normalization rule that you create with the MD Editor (refer to the procedure To create a new normalization rule file). Once you have created the process, you can, subsequently, run a job using that process (see Running Manual Jobs on Defined Sets).
    Normalization rules are created by following a specific programming syntax and using the editing window provided in the MD Editor under the Rules tab.
    Rules_Tab_Editing_Window_NewUI_02_TC.png
    Rules Tab Editing Window
    In addition to creating your own original normalization rules program, you can also copy/paste existing rules into the editing window or use out-of-the-box examples (Rules tab > Edit > Add Rule) to develop your normalization rules. See Normalization Rules Syntax for more information regarding the syntax for normalization rules and examples that you can copy to the editing window.
    Using the preview functionality in the MD Editor, you can:
    • View the normalization rules and metadata records side by side
    • Preview the outcome of a rule file when run on a metadata record
    • Toggle between the rule file and the preview changes
    • Edit rules and test immediately
    This section describes how to create a normalization rule file, work with previously created rule files, and preview the outcome of a rule file when it is run on a single metadata record. For additional information on normalization rules, see Normalization Rules and Processes (.pptx file).
    For a video on Normalization rules, see Normalization Rules (41:07 mins). For a detailed Ask the Expert session on configuring normalization rules, see Normalization Rules.
    To create a new normalization rule file:
    1. On the MD Editor page (Resources > Cataloging > Open Metadata Editor), to create a normalization rule for a MARC record, select File > New > Normalization Rule and to create a normalization rule for a DC record, select File > New > XSL Transformers. The Normalization Rules Properties dialog box opens.
      Normalization Rules Properties Dialog Box
    2. Enter a name (required) and a description for the normalization rule file.
    3. Select an access option, Private or Shared. If you select the former, only you can work on the rule and the rule cannot be included in a normalization process. If you select the latter, your rule will be shared among catalogers. In this case, more than one user can view the rule at the same time, and if two or more people have the rule open for editing, a warning message appears when one of you tries to save changes. (You have the option of keeping your changes or allowing the other user to make and save changes.)
    4. Select Save. A text box, in which you enter the rule opens.
      You can include existing rule syntax (Edit > Add Rule > {type of rule}) or define a rule (for details, see Normalization Rules Syntax).
    5. Select Save. The rule file is added to the list of rule files in the Normalization Rules tab.
    To work with an existing normalization rule file:
    1. On the MD Editor page (Resources > Cataloging > Open Metadata Editor), For a MARC normalization rule, select the Rules tab and expand the Normalization rules folder to display the saved rule files and for DC normalization rules, select the XSL Transformers tab.
    2. Select the rule file with which you want to work and select one of the following options:
      • Edit – Opens the text box with the rule(s) syntax, enabling you to modify this syntax (for details, see Normalization Rules Syntax).
      • Delete – Select Yes to confirm the rule file’s deletion.
      • Duplicate – Duplicates the selected rule file, enabling you to modify and save it as a new rule file without affecting the original file.
      • Properties – Opens the Normalization Rules Properties dialog box, enabling you to modify the properties of the rule file.
    To preview the outcome of a rule file:
    1. Locate the bibliographic record with which you want to work (using the Repository Search or within the MD Editor Records tab) and open it in the MD Editor.
    2. Select Edit > Split Editor (F6) or select the Split Editor icon.
    3. Select the Rules tab, expand the Normalization rules > Shared list to display the saved rule files, select the rule file whose outcome you want to preview, and select Edit.
      Rules_Actions_NewUI_02_TC.png
      Select Edit to Open the Normalization Rule
      The rule file is displayed in the right pane of the MD Editor.
      Normalization_Rule_Preview_14.png
      Normalization Rule Preview
    4. Select Preview. The rule in the file is applied to the record and the outcome appears.
      When previewing a rule for a Network record, the following message is displayed:
      Note that the rules will be applied only to local fields when the normalization process is run.
    5. Select Apply Changes to save the modifications to the record or select Back to normalization rules.
    If you return back to your normalization rule, you can make additional changes to your normalization rule and, iteratively, select Preview and view your changes. When you have made your final changes to the normalization rule, select Save (next to the Preview button) to save the final version of your normalization rule.

    Normalization Rule Syntax

    Rule files contain one or more rules, which contain a condition and one or more actions to be applied to records. Actions are applied to a record if the record meets the condition. Each action within a rule can be performed on a single field within a record. Actions are performed in the order in which they appear within the rule.
    When defining a rule that contains multiple rules, you must use the priority (or salience) factor. Actions stipulated in the rule with the higher priority are performed first. For example, the action in a rule with priority 2 is performed before the action in a rule with priority 1. For examples of the way in which priority (or salience) can be used in normalization rules, see Normalization Rule Examples
    Note that priority (or salience) is only necessary when the fields in the rules are the same. Otherwise, when the rules in a file deal with different fields, multiple rules are performed from top to bottom.
    Learn more about creating normalization rules in the Normalization Rules video (41:07 mins).
    Learn how to create normalization rules that delete specified fields from records, or change the contents of these fields in the Normalization Routine Syntax for Deletion or Content Change video (9:57 mins).
    When
    (<conditions on MARC record>) then
    Action1
    Action2 if condition
    Action3
    End
    <conditions on MARC record> contains one or more Boolean clauses that apply to the record. If <conditions on MARC record> returns TRUE, the rule is applied to the record; otherwise, the rule is not applied and the record is not processed.
    • “When” must be the only word in the first line. The condition must be placed on a separate line.
    • While it is permissible to include multiple Boolean Operators in the rules, when a large number of Boolean operators are selected, slower performance is likely to result. Thus each rule should include no more than 200 Boolean operators.

    Record Elements

    Conditions and actions apply to record elements, such as the MARC record, fields (one or more), indicators, subfields (one or more), and field/subfield contents.
    To test a condition or apply an action to a record element, the element must match the following syntax:
    Syntax
    Expression Meaning
    "<tag>", "<new tag>" Represents a field tag, for example, 001, 245, etc.
    "<oldCode>", "<newCode>" Represents a subfield code, for example, a, b, c.
    "<element>" for a data field The following are the possible values for the data field:
    • FIELD– for example: 245
    • FIELD_VALUE – for example: 245.value*
    • FIELD_INDICATOR – for example: 245.{1,2}
    • FIELD_SUBFIELD_CODE – for example: 245.a
    • FIELD_INDICATOR_SUBFIELD_CODE – for example: 245.{1,2}.a
    • FIELD_SUBFIELD_CODE_VALUE – for example: 245.a.value*
    • FIELD_INDICATOR_SUBFIELD_CODE_VALUE – for example: 245.{1,2}.a.value*
    "<element>" for a control field The following are the possible values for a control field:
    • FIELD_POSITION_LENGTH – for example: LDR.{17,3}
    • FIELD_POSITION_LENGTH_VALUE – for example: LDR.{17,3}.eng
    CONDITION at record level The following are the possible condition options. See the next section (Conditions) for important information.
    • TRUE – always true
    • not exists "{element}" – true if the element does not exist (for data fields)
    • not existsControl "{element}" – true if the element does not exist (for control fields)
    • exists "{element}" – true if the element exists at least once (for data fields)
    • existsControl "{element}" – true if the element exists at least once (for control fields)
    • existsMoreThanOnce "{element}" – true if the element exists more than once. See Managing Search Queries and Sets for an example of the use of this condition.
    • contains – true if the element contains a value. Used for merge rules.

    Conditions

    Conditions can be defined at the entire rule level (WHEN), or at a specific action level (IF). The same condition will behave differently depending on the level at which it is defined.
    • WHEN clause – A condition that must be met by the entire record in order to determine whether the rule is applied to the record
    • IF (within an action) – A condition that applies to a single field in order to determine whether the specific action is taken on that field
    Conditions can be:
    • exists <element> – At least one match is found
      • exists <element> – Applies to data fields. When used in an IF clause, both the action element and the element tested by the condition must be the same (data) field.
      • existsControl <element> – Applies to control fields. When used in an IF clause, both the action element and the element tested by the condition must be the same (control) field.
    • existsMoreThanOnce <element> – Multiple matches are found. Applies to data fields. When used in an IF clause, both the action element and the element tested by the condition must be the same (data) field.
    • not exists <element> – No match is found
      • not exists <element> – Applies to data fields. When used in an IF clause, both the action element and the element tested by the condition must be the same (data) field.
      • not existsControl <element> – Applies to control fields. When used in an IF clause, both the action element and the element tested by the condition must be the same (control) field.
    • recordHasDuplicateSubfields (for indication rules; see Working with Indication Rules) – Returns true if duplicate subfields (subfield and its contents) are found for the current record according to the fields, subfields, and the characters to ignore (charsToIgnore) string that were passed as parameters in the following format:
      recordHasDuplicateSubfields "<tag>" "<code>" "<charsToIgnore>"
      Multiple tags (fields) separated by commas can be specified. Multiple codes (subfields) can be specified with no spaces to separate them. One or more characters (alphanumeric or punctuation) with no spaces to separate them can be specified as characters to ignore at the end of the content in the subfields that are being evaluated for duplication. See Example 6 for more information.
      For the records that meet the recordHasDuplicateSubfields condition (returns true), a set of the records is created.
    Each IF clause action can have one of the following formats:
    • Applies if a specific condition is not true, for example: addControlField "{element}" if(not exists "{condition}")
    • Applies if a specific condition is true, for example: addControlField "{element}" if(exists "{condition}")
    • Applies unconditionally, for example: addControlField "{element}"
    The Boolean OR operator can be used in a consequence statement by using the pipe (|) symbol, for example: removefField "866" if (not exists "866.8.0|99")
    If the pipe symbol is part of the value, use four backslashes (\\\\) to escape it, for example: removeField "866" if (exists "866.8.0\\\\|99")

    List of Actions

    The following table provides a list of available actions.
    List of Actions
    Action Format / Example Comment
    Replace fields and subfields with other fields and subfields changeControlField "<tag>" to "<new tag>"
    Example: changeControlField "007" to "008"
    Changes the tag identifier of a control field; does not modify contents.
    changeField "<tag>" to "<new tag>"
    Example: changeField "245" to "246"
    Changes the tag identifier; does not modify indicators or subfields.
    changeSubField "<tag>.<code>" to "<new code>"
    changeSubFieldOnlyFirst "<tag>.<code>" to "<new code>"
    changeSubFieldExceptFirst "<tag>.<code>" to "<new code>"
    Example: changeSubField "035.b" to "a"
    Changes the subfields (or only the first subfield, or all except the first subfield) "<code>" to the subfield "<new code>" in field "<tag>".
    changeFirstIndicator "<tag>" to "<value>"
    changeSecondIndicator "<tag>" to "<value>"
    Example: changeFirstIndicator "245" to "3"
    Sets the value of the specified indicator in tag <tag>.
    combineFields "<tag>" excluding "<comma-separated subfield list>"
    Example: combineFields "852" excluding "a,b"
    Combine all fields of the specified number. Copy all subfields from the second and subsequent lines to the first line, excluding the named subfields; only the first occurrences of excluded subfields are copied, and only if they do not already exist in the first line.
    Add fields and subfields addField "<tag>.<code>.<value>"
    addField "<tag>.{<ind1>,<ind2>}.<code>. <value>"
    Example: addField "999.a.RESTRICTED"
    Adds the field to the MARC record. Sets the value of the subfield to the indicated value.
    addControlField "<tag>.<value>"
    Example: addControlField "008.820305s1991####nyu###########001#0#eng##"
    Adds the control field to the MARC record.
    addSubField "<tag>.<code>.<value>"
    addSubField "<tag>.{<ind1>,<ind2>}.<code>.<value>"
    Example: addSubField "245.h.[Journal]"
    Adds the subfield <code> with value <value> to field <tag>. If the field does not exist, nothing is done.
    addSystemNumber "<element>" from "<tag>" prefixed by "<prefix tag>"
    Example: addSystemNumber "035.a" from "001" prefixed by "003"
    Makes the data field <element> equal to the contents of the second control field <prefix tag> in parentheses followed by the contents of the first control field <tag>.
    For example, if 001 has the value 9945110100121 and 003 has the value DAV, the example condition on the left will produce 035 with the value ‡(DAV)9945110100121.
    Copy fields copyField "<tag>" to "<new tag>"
    copyField "<tag>.<code>" to "<new tag>.<new code>"
    copyField "<tag>" to "<new tag>.{<ind1>,<ind2>}"
    Example: copyField "971.a" to "100.u"
    Copies the field to another field. In the first version, the subfields are not specified (<code> and <new code>), and the new field contains all the same subfields as the old field. In the second version, if just <new code> is not specified, the new subfield is the same as the one specified by <code>.
    copyField creates a separate field rather than adding it to any existing field. You may want to combine the new field with any existing fields (see combineFields).
    Remove fields and subfields removeControlField "<tag>"
    Example: removeControlField "009"
    Removes all occurrences of the control field.
    removeField "<tag>"
    Example: removeField "880"
    Removes all occurrences of the field <tag>.
    removeSubField "<tag>.<code>"
    Example: removeSubField "245.h"
    Removes all occurrences of the subfield <code> from the indicated field.
    Replace text in fields or subfields replaceControlContents "<tag>.{<position>,<length>}.
    <value>" with "<new value>"
    Example: replaceControlContents "LDR.{7,1}.s" with "m"
    Replaces <value> with "<new value>" in starting position <position> to <position>+<length> of control field <tag>. Replaces only the text that matches <value>.
    replaceContents "<tag>.<code>.<value>" with "<new value>"
    replaceContentsOnlyFirst "<tag>.<code>.<value>" with "<new value>"
    replaceContentsExceptFirst "<tag>.<code>.<value>" with "<new value>"
    Example: replaceContents "245.h.[Journal]" with "[Book]"
    Replaces the matching strings (or every instance of the matching string in the first matching subfield, or all matching strings in all matching subfields except the first matching subfield) <value> in the subfield <code> of field "<tag>" with "<new value>". The string or part of the string that does not match <value> is not modified.
    replaceSubFieldContents "<tag>.<code>" with "<tag>.<code>"
    Example: replaceSubFieldContents "245.b" with "100.a"
    Replaces the subfield's contents with the contents of another subfield.
    Add text in subfields
     
    prefix "<tag>.<code>" with "<value>"
    Example: prefix "035.b" with "(OCoLC)"
    Adds a prefix to the value of subfield "<code>" in the field "<tag>".
    The new value will be <value> followed by the old value.
    prefixSubField "<tag>.<code>" with "<source tag>.<source code>"
    Example: prefixSubField "910.a" with "906.a"
    Adds the value of the subfield "<source code>" in the field "<source tag>" as a prefix to the subfield "<code>" in the field "<tag>".
    The new value will be the value of the subfield "<source code>" in the field "<source tag>" followed by the old value.
    suffix "<tag>.<code>" with "<value>"
    Example: suffix "035.b" with "(OCoLC)"
    Adds a suffix to the value of subfield "<code>" in the field "<tag>".
    The new value will be the old value followed by <value>.
    suffixSubField "<tag>.<code>" with "<source tag>.<source code>"
    Example: suffixSubField "910.a" with "907.c"
    Adds the value of the subfield "<source code>" in the field "<source tag>" as a suffix to the subfield "<code>" in the field "<tag>".
    The new value will be the old value followed by the value of the subfield "<source code>" in the field "<source tag>".
    Maintain agency information in bibliographic and authority records
    For example, this syntax can be used in normalization rules that are selected in the MARC 21 Bibliographic Metadata Configuration Task List to normalize Network Zone bibliographic records upon save.
    This functionality is under construction. To enable this syntax, contact Ex Libris Support.
    addCreatingAgency "<tag>.<code>"
    Example: addCreatingAgency "040.a"
    Adds the creating agency ISIL code to the subfield in "<code>" in the field "<tag>".
    addModifyingAgency "<tag>.<code>"
    Example: addModifyingAgency "040.d"
    Adds the modifying agency ISIL code to the subfield in "<code>" in the field "<tag>". If there already is a modifying agency in the "<tag>.<code>", this adds another agency ISIL code.
    replaceModifyingAgency "<tag>.<code>"
    Example: replaceModifyingAgency "040.d"
    Adds the modifying agency ISIL code to the subfield in "<code>" in the field "<tag>". If any modifying agencies already exist in the "<tag>.<code>", all of them are replaced.
    Split subfields splitSubField "<tag>.{ind1,ind2}.<code>.<delimiter>" to "<tag>.{<ind1>,<ind2>}.<code>" addSeq "<code>"
    Example 1: splitSubField "866.a.;" to "555.{0,0}.a" addSeq "8"
    Example 2: splitSubField "555.a. – " to "859.{0,0}.a" addSeq "8"
    Example 3: splitSubField "859.a.\\\\."
    Example 4: splitSubField "999.a.;" to "555.a" addSeq "8"
    The tag is mandatory.
    The indicators are optional.
    Since the split is at the subfield level, the code is mandatory.
    The delimiter can be any string. If the delimiter does not exist, the full subfield is copied as the first (and only) occurrence, and the sequence is added.
    The to component is optional. If it is specified, multiple occurrences of the to tag.code are created, each with the data until the delimiter. See examples 1 and 2. If the to component is not specified, the subfield is split to the additional same subfields in the same field as shown in example 3.
    The addSeq component is optional. It is not relevant if the to component is not specified. When addSeq is specified, the subfield with a sequence will be added as in Example 1; and if the subfield already exists in the original field, a sequence (preceded by a period) is added to that field as in Example 2.

    Remove duplicate subfields

    correctDuplicateSubfields "<tag>" "<code>"

    Example: 

    Removes duplicate subfields x, y and z from fields 610 and 630. 
    rule "Remove duplicates" 
    priority 1 
    when 
    (TRUE) 
    then 
    correctDuplicateSubfields "610,630" "xyz" 
    end

    Corrects duplicate subfields by keeping the first occurrence and removing the others from the current record according to the fields and subfields that were passed as parameters.

    You may want to use recordHasDuplicateSubfields to create the set that you supply to your normalization rule that uses correctDuplicateSubfields. See Example 6 for more information.

    Move subfields

    moveSubfieldsToEndOfField "<tag>" "<code>"

    Example: 

    Moves subfields 9 and 2 to the end of field 650.
    rule "Move subfields to end of field" 
    priority 1 
    when 
    (TRUE) 
    then 
    moveSubfieldsToEndOfField "650" "92" 
    end

    Moves the first occurrence of each subfield to the end of the field and removes all other occurrences of the same subfield.

    If more than one subfield is specified, they are placed at the end in the same sequence as identified in the rule. In this example, subfield 9 is placed at the end followed by subfield 2.

    Wildcards and Special Characters

    The subfield separator ($$) cannot appear anywhere in the rule, including the rule name.
    A hash character (#) at the beginning of a line indicates that the rest of the line is a comment, and is ignored when processing the rule.
    The asterisk (*) is used to match any string, including a string of zero length. For example, "<tag>.<*>.<value>" applies to all the subfields in the tag <tag> that have the value <value>. * is "greedy", so it matches as many characters as possible in the string. For example: if you have a string "a b c b d b e", the pattern "b*b" matches "b c b d b", not just "b c b".
    Empty indicators (but not fields or subfields) are indicated by a hyphen (-). For example, "<tag> {-,<ind2>}" returns all the fields where the MARC tag is <tag>, the first indicator is not defined, and the second indicator is <ind2>.
    If the text of a subfield contains a period as the last character, use four backslashes to match the period. For example:
    rule "Replace '1 v.' to 'Leaves' in $a (unconditional)"
    when
    (TRUE)
    then
    replaceContents "300.a.1 v\\\\." with "Leaves"
    end
    Double quotation marks can be used in conditions (only). To use double quotation marks as part of a condition, use single quotation marks to enclose text in the rule (') instead of double quotation marks ("). This way you can use double quotation marks inside the text following a double backslash (\\).
    rule "populate 008 7-10 2016"
    when
    (exists '245.{*, }.c.\\"')
    then
    replaceControlContents "008.{7,4}" with "double quote"
    end
    Hebrew dates can be used in conditions (only). (Hebrew reads from right to left, so the double backslashes in the following example are actually before the double quotes.)
    rule "populate 008 7-10 2016"
    when
    ((exists '260.{*, }.c.תשע\\"ו') OR (exists '264.{*, }.c.תשע\\"ו'))
    then
    replaceControlContents "008.{7,4}" with "2016"
    end
    • Wildcards cannot be used as the first character of a condition or value.
    • To use a literal backslash (\), escape it with another backslash: \\. Similarly, to use a literal asterisk, escape it with a backslash.
    • Use four backslashes (\\\\) to escape a period when it is the last character in the string. When the period is immediately followed by another character, it does not require four backslashes (as in addField "907.a.F.L.T\\\\."). However, it is best practice to always use the four backslashes in the normalization rule to ensure the most consistent desired results. See the following examples.
    • As noted above, if a double quote is used in a condition (only) that is escaped using single quotes, or a single quote is used in a condition that is escaped using single quotes, you must further escape the double quote or quote with a double backslash.
    • If the pipe symbol is part of the condition, use four backslashes to escape it, for example: removeField "866" if (exists "866.8.0\\\\|99"). This is required only when using a pipe symbol in the condition.

    Example: Using the Period in a Normalization Rule with replaceContents

    Example record with periods:
    245 00 $$a Feminist literary theory. : $$b a reader / $$c edited by Mary Eagleton.
    246 0# $$a F.L.T.
    Normalization rule for the example record above:
    rule "remove the periods in 245 and 246 subfield a (and replace periods with nothing); precede period with four backslashes"
    when
    (TRUE)
    then
    replaceContents "245.a.\\\\." with ""
    replaceContents "246.a.\\\\." with ""
    end
    See the figure below for the before and after examples.
    Before and After Examples

    Example: Using the Period in a Normalization Rule with addField

    Below is an example of a record to which periods need to be added:
    906 $$a Architecture.
    907 $$a F.L.T.
    Below is the normalization rule for the example record above using the best practice of always including slashes:
    rule "Add field 906 with text Architecture and period at end and also add field 907 with F.L.T."
    salience 100
    when
    TRUE
    then
    addField "906.a.Architecture\\\\."
    addField "907.a.F\\\\.L\\\\.T\\\\."
    end
    See the figure below for the before and after record examples.
    Before and After Examples

    Normalization Rule Examples

    For a list of over 50 normalization rule examples and other normalization rule documents, see Normalization Rules.