Natural Language Search in the NDE UI

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

Introduction

The Natural Language Search feature enables users to formulate queries in normal spoken language, and automatically converts them into the structured format compatible with Primo’s Advanced Search. For example, the user could enter the query, "Find me US history journals in English that are available online," and the system would create a query with the appropriate criteria for the search.

Not only does the Natural Language Search generate queries from free text, it also identifies certain catch words in the text that can be used to define the scope and automatically select the appropriate filters for the search. For example, if the term "journal" appears in the text, the scope of the search is limited to journals, and if a language is specified, the language filter is automatically turned on.

The transformation of the original query into the structured format is performed using generative AI, via ChatGPT 4.1Mini. The elements of the query, such as resource types, date filters, full text preferences, language, and advanced fields like Title or Subject, are classified by the AI system, which uses this information to generate the basic Boolean query and then expands it using related concepts. Ambiguous inputs are handled intelligently by mapping to multiple fields (e.g., both Title and Subject) to ensure broad yet accurate retrieval.

An internal preventive measure limits the number of Natural Language Search calls per site, so that we can control the overall costs and avoid abuse of the system.

Performing a Natural Language Search

Natural Language Search is enabled per View. When it is enabled, an Ask Anything button appears beside the Search box. When the user selects the button, the Ask Anything page opens in a sliding panel.

Ask Anything Page open in a sliding Panel.

Ask Anything Page

To enter a search query in the Ask Anything page:

Under Show results from, select the scope of the search.
In the text box, enter a query.
Press Enter or select Search. The query is converted into a Boolean query, the panel closes, and the brief results open. A bar above the brief results shows the search query and provides editing options.

Editing the Search Query

Once the search has been performed, you can edit it in one of the following ways:

Select Edit to reopen the Ask Anything page and modify the scope and/or the query.
Select Simple Search to close the Ask Anything bar and revert to the standard Simple Search options.
Select Generated Query to see the query in the form in which it was sent to the search engine. The Advanced Search page opens in a sliding panel and shows the modified query. The query can be edited in the page and then run again by selecting Search.

Activating Natural Language Search

Natural Language Search is activated per View in the View configuration. It is activated by default, but can be deactivated in the settings.

To activate/deactivate Natural Language Search:

In the View Configuration page (Discovery > Display Configuration > Configure Views > [Edit a View]), in the General tab, select or clear the Enable Natural Search checkbox.

Special Terms

In the process of transforming a free-text query into a Boolean query, the system identifies special terms that can be used to define the fields of the query and to automatically apply filters and facets to the search results, as follows:

Advanced search fields:

Indexes – ISSN, ISBN, MMS_ID, OCLC
Author
Title-related fields – Title, Description, General, Table of Contents

Filters and facets:

Resource Types
Top level facets – Available Online, Peer-reviewed, Open Access, Held by library.
Creation Date
Requested Language

Search History

If the save_users_search_history Customer Parameter is set to true (Configuration > Discovery > Other > Customer Parameters), the system retains the properties of the previous Natural Language searches performed by the user. The last three saved searches appear in the Ask Anything page below the text box. The user can repeat one of the searches by selecting it, which copies its properties into the search fields above the list. They can also delete individual past searches if they wish. When a past search is deleted, the search performed immediately before it appears in the list.

Analytics

Data about user interactions with Natural Language Search is available for tracking and analysis through Mixpanel. For information about the available data, see Mixpanel - Events and Properties for the NDE UI.

Known Issues

The following are known issues that we are currently working to resolve:

When the language is changed to a non-English language, the language used in the generated Boolean query is not always changed accordingly.
When the language is not English, part of the identified terms are sometimes also included in the Boolean query (when they should be omitted).
In cases where identifiers or authors are detected, the relation between the different fields may appear as OR (except for the first variation) instead of AND, which results in expanded search results.
In some cases, the Author field is not identified as expected.

Clarivate's Approach to Addressing Environmental Impact through AI Tools

The conversation regarding carbon emissions caused by AI is part of a wider discussion around systems, data storage and sustainability which cannot be solved by any one organization. This is an important industry-wide challenge, which we are working with our vendors, customers and community to understand and address.

By continuously focusing on actions and outcomes at Clarivate, we are making a positive impact on our business, our people and our planet. We are mindful of the need to reduce waste in systems and data storage and are committed to get to net zero carbon emissions before 2040. Our products and services are designed, developed and deployed following environmental and sustainable best practices including optimizing to reduce waste and pollution such as CO2 emissions. We are building a comprehensive climate transition plan that includes setting Science Based Targets (SBTs).

We are working in close partnership with all our cloud systems and data storage providers (Amazon, Google and Microsoft.) Each of these companies has their own sustainability commitments and ambitions to get to net zero by 2030 or 2040, the majority of which is aimed at reducing emissions created in the first place through using green energy rather than offsetting.

Our Environmental Management Statement outlines our framework to adjust existing working practices towards our Net Zero before 2040 target, which includes our usage of AI services. For more information, please see our Sustainability Report.

In academia, our Academic AI platform serves as a technology backbone, enabling a centralized and consistent deployment of AI capabilities across our portfolio of solutions and promoting efficient performance. We choose AI models that balance performance, cost, and sustainability. For each task, we always prioritize high-quality output with the least resource-intensive technology.

Our centralized platform approach also helps eliminate system redundancies, reducing both resource consumption and emissions. Additionally, we use caching and text compression mechanisms to reduce the workload on the LLM, making LLM calls more efficient.

We continue to engage with stakeholders across academic communities to explore and implement best practices for reducing the environmental impact of AI.

AI Limitations, Warnings, and Guardrails

While powerful, LLMs have limitations and may generate inaccurate responses. We recommend users check for accuracy and verify the responses against the source materials provided.

For more information about Data Privacy Protection and the use of Large Language Models (LLMs), see Clarivate Academia & Government - Use of Generative AI.

For information about how Clarivate supports safe, ethical use of generative AI in academic research through responsible AI guardrails and content filtering, see Guardrails for Responsible AI: Balancing Safety and Academic Discourse.