[BUG] Hybrid Search Variant ids are incorrectly created

[alexeyrodriguez]

What is the bug?

A Hybrid Search experiment should evaluate 66 Hybrid search variants. If there is a query set with 2 queries, the total number of results generated in the search-relevance-evaluation-result index should be 123 (66 x 2). Unfortunately there are too many experiment variant ids generated, the total number of generated variant ids ends up being 123 rather than 66.

How can one reproduce the bug?

Run the hybrid experiment from the demo script and run the following query using the id of the created experiment:

GET /search-relevance-experiment-variant/_search
{
  "size": 0,
  "query": {
    "match": {
      "experimentId": "EXPERIMENT_ID"
    }
  },
  "aggs": {
    "unique_experiment_variant_ids": {
      "cardinality": {
        "field": "id"
      }
    }
  }
}

Result:

{
  "took": 3,
  "timed_out": false,
  "terminated_early": true,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 132,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "unique_experiment_variant_ids": {
      "value": 132
    }
  }
}

What is the expected behavior?

The above query such return 66 unique values rather than 123.

What is your host/environment?

n/a

Do you have any screenshots?

n/a

Do you have any additional context?

n/a

[martin-gaievski]

This behavior is by design, in #29 the concept is to combine set of parameters and actual runtime data for each variant. There is one experiment record for each evaluation we run, and because we evaluate on each query text this means one variant for each combination of parameters + query text + search configuration.
We discussed offline with @alexeyrodriguez possible ways to address this, two options are on the table:

1. use opposite approach and store references to experiment and variant in the evaluation document
2. keep existing structure but make the result field of variant a list (today it's a single object)

Option 1 is more frontend friendly and will make experiment variant object cleaner by keeping it responsible for experiment configuration only.
Option 2 giving us more flexibility in experiment setup (different query text may use different set of variants), but that's not need at the moment.

[wrigleyDan]

I think I misunderstood the data structure we have until now.

I had the understanding that for hybrid search optimization the following is correct:

• Every hybrid search optimization experiment receives a unique id
• Every hybrid search optimization experiment creates n experiment variants according to the number of parameter combinations that are experimented with. Currently, in 3.1 this number is 66: 11 weight options * 2 normalization options * 3 combination options. This results in n experiment variant ids
• Every experiment variant runs x queries (equals size of corresponding query set) and thus creates n times x evaluations. For a query set size of 2 this should result in 132 evaluation ids: 66 experiment variants are tried for 2 queries.

From what I understand now the data structure is as follows:

• Every hybrid search optimization experiment receives a unique id
• Every hybrid search optimization experiment creates n times x experiment variants. n is the number of parameter combinations, x is the number of queries.
• The result of each variant id is in the evaluations index and receives its own id.

I think my misunderstanding mainly results from the fact that there is no notion of the query in the experiment variant document which I interpreted incorrectly.

If I understand option 1 correctly, option 1 would be my preference as it would align with my initial understanding:

• remove the result from experiment variant and only store one variant per parameter combination, not one per parameter combination per query
• create links to experiment and variant in the evaluation docs, so it becomes evident when looking at the evaluations index to which experiment and variant the result belongs.

Is my understanding here correct?

Whatever way we choose to move forward: this needs more documentation if we within the core dev group have difficulties understanding the internals.

[alexeyrodriguez]

@wrigleyDan I would also agree that solution 1 is to be preferred. As it happens, the Dashboards work already requires the variant id to be in the results document, this is implemented in the following PR: #174

Next, I made the following change to the UI, so that results retrieval is made using the id fields that were added to the result documents: #174 . This PR would allow @martin-gaievski to fix the variant id generation so that there are only as many variant ids as configurations without breaking the UI retrieving logic.