Allow thresholding on vector and fulltext indexes for Hybrid retrievers

[willtai]

Description

Allow thresholding on vector and fulltext indexes for Hybrid retrievers. Two thresholds can be provided by the user during search to determine the importance of the search results from either vector or fulltext index.

Type of Change

• New feature
• Bug fix
• Breaking change
• Documentation update
• Project configuration change

Complexity

Complexity: Low

How Has This Been Tested?

• Unit tests
• E2E tests
• Manual tests

Checklist

The following requirements should have been met (depending on the changes in the branch):

• Documentation has been updated
• Unit tests have been updated
• E2E tests have been updated
• Examples have been updated
• New files have copyright header
• CLA (https://neo4j.com/developer/cla/) has been signed
• CHANGELOG.md updated if appropriate

[alexthomas93]

LGTM, just a few minor points

[stellasia]

I'd be interested in seeing an example of this.

Here are the points bothering me:

• Aren't these parameters too query-dependent, since the range of normalized scores can vary a lot for each query?
• What if most of the scores end up being 0? Then we do not have any ordering?
• Is this compatible with the effective search ratio implemented in langchain? Typically, this ratio would bring more items in this part of the query and so change the normalization.

[willtai]

@CodiumAI-Agent /update_changelog

[CodiumAI-Agent]

Changelog updates: 🔄

2025-01-15

Added

• Support for thresholding on vector and fulltext indexes in Hybrid retrievers, enabling users to set importance levels for search results.

to commit the new content to the CHANGELOG.md file, please type:
'/update_changelog --pr_update_changelog.push_changelog_changes=true'

[alexthomas93]

Why change this to a multi-line string here?

[willtai]

This helps me modify the unit tests more easily without worrying about correct indentation and spaces

[alexthomas93]

With multi-line strings, you do have to worry about correct indentation though. With the

(
"Hello "
"world"
)

approach you only have to worry about there being a space at the end of every line.

[willtai]

Hmm I still generally prefer multi-line strings as I find them more readable and tend to avoid mistakes after changes. Do you think we should revert this back?

[alexthomas93]

I’m inclined to prefer the older approach, but I’m also open to this option. @stellasia what are your thoughts?

[NathalieCharbel]

I apologise for chiming in a bit late, but here's my opinion.
While thresholding in a hybrid retriever can be useful, it may also introduce nuances that are misaligned with the main objective of identifying the most relevant results from both full-text and vector-based searches (with thresholding, you often filter out any results that do not exceed a certain threshold in one search method before considering their relevance in the other method, so relevant results can be lost prematurely).
Also, from a user's perspective, I find that adjusting how much weight is given to each type of score is more intuitive than tuning two threshold parameters based on score, which might be an obscure value for the user.
One way to do this is by combining the vector and full-text scores into a single one. So in general, you fetch top candidates from each search, compute one combined score, and then pick the best overall. This way the user only needs to tune a single parameter, something like:
combined_score = α * score_vector + (1−α) * score_fulltext where α ∈ [0,1]
wdyt?

[CodiumAI-Agent]

Changelog updates: 🔄

2025-01-22

Added

• Support for thresholding on vector and fulltext indexes in Hybrid retrievers, enabling users to set importance levels for search results.

to commit the new content to the CHANGELOG.md file, please type:
'/update_changelog --pr_update_changelog.push_changelog_changes=true'

[CodiumAI-Agent]

Changelog updates: 🔄

2025-02-24

Added

• Support for thresholding on vector and fulltext indexes in Hybrid retrievers, enabling users to set importance levels for search results.

to commit the new content to the CHANGELOG.md file, please type:
'/update_changelog --pr_update_changelog.push_changelog_changes=true'