Skip to content

[BUG] Unnecessary index creation actions when creating experiment #156

Open
@martin-gaievski

Description

@martin-gaievski

What is the bug?

Behind the scenes there are multiple concurrent attempts to create SRW specific indexes internally. This waists system resources and may put system stability at risk.

How can one reproduce the bug?

It's visible in service logs in production scenario but easier to repro using tests

  1. Change SearchRelevanceIndicesManager class by adding logging
 private void createIndexIfAbsentSync(final SearchRelevanceIndices index) {
        String indexName = index.getIndexName();
        String mapping = index.getMapping();

        if (clusterService.state().metadata().hasIndex(indexName)) {
            log.debug("Index [{}] already exists, skipping creation", indexName);
            return;
        }
        final CreateIndexRequest createIndexRequest = new CreateIndexRequest(indexName).mapping(mapping);
        StashedThreadContext.run(client, () -> client.admin().indices().create(createIndexRequest, new ActionListener<>() {
            @Override
            public void onResponse(final CreateIndexResponse createIndexResponse) {
                log.info("Successfully created index [{}]", indexName);
            }

            @Override
            public void onFailure(final Exception e) {
                if (e instanceof ResourceAlreadyExistsException) {
                    log.info("index[{}] already exist", indexName);
                    return;
                }
                log.error("Failed to create index [{}]", indexName, e);
            }
        }));
    }
  1. Run integration tests with command gradle integTest
  2. In server logs for test cluster observer following messages:
[2025-06-25T13:05:43,608][INFO ][o.o.p.PluginsService     ] [integTest-0] PluginService:onIndexModule index:[search-relevance-evaluation-result/3iphp1HaSDWEbQPFoIEgZA]
[2025-06-25T13:05:43,633][INFO ][o.o.s.i.SearchRelevanceIndicesManager] [integTest-0] index[search-relevance-evaluation-result] already exist
[2025-06-25T13:05:43,634][INFO ][o.o.s.i.SearchRelevanceIndicesManager] [integTest-0] index[search-relevance-evaluation-result] already exist
[2025-06-25T13:05:43,635][INFO ][o.o.s.i.SearchRelevanceIndicesManager] [integTest-0] index[search-relevance-evaluation-result] already exist
[2025-06-25T13:05:43,761][INFO ][o.o.s.i.SearchRelevanceIndicesManager] [integTest-0] Successfully created index [search-relevance-experiment-variant]
[2025-06-25T13:05:43,885][INFO ][o.o.s.i.SearchRelevanceIndicesManager] [integTest-0] Successfully created index [search-relevance-evaluation-result]
[2025-06-25T13:05:45,733][INFO ][o.o.s.i.SearchRelevanceIndicesManager] [integTest-0] index[search-relevance-experiment-variant] already exist
[2025-06-25T13:05:45,744][INFO ][o.o.s.i.SearchRelevanceIndicesManager] [integTest-0] index[search-relevance-experiment-variant] already exist

For search-relevance-experiment-variant index this is especially bad, for me there were 620 messages about failed index creation

What is the expected behavior?

We try to create index once at best, few failing attempts are still fine. Don't create index on next run when it's already present in the system.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    Status

    Not Committed

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions