[high cardinality queries] track OOM on trace item table and time series endpoints

[xurui-c]

https://github.com/getsentry/projects/issues/463

[sentry-io]

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: snuba/clickhouse/native.py

Function	Unhandled Issue
execute	ClickhouseError: Connection refused (127.0.0.1:9058) clickhouse_n... Event Count: 5

---

_{Did you find this useful? React with a 👍 or 👎}

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
2795	1	2794	11

View the top 1 failed tests by shortest run time

tests.web.rpc.v1.test_endpoint_time_series.test_endpoint_time_series.TestTimeSeriesApi::test_OOM

Stack Traces | 0.27s run time

Traceback (most recent call last):
  File ".../v1/test_endpoint_time_series/test_endpoint_time_series.py", line 907, in test_OOM
    sentry_sdk_mock.assert_called_once()
  File ".../local/lib/python3.11/unittest/mock.py", line 918, in assert_called_once
    raise AssertionError(msg)
AssertionError: Expected 'capture_exception' to have been called once. Called 0 times.

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

[phacops]

Do we really need to match on the code and the string? Or do we need to look for the full message as opposed to just Memory limit?

[xurui-c]

We can do either. I'm kind of making the assumption here that ClickHouse's OOM error will always be 241. As an extra measure I'm also doing string matching with "DB::EXception: Memory..."

I'm not string matching for the full message since the full message can be extremely long and differs for every error, but it seems like "DB::Exception: Memory limit (for query) exceeded" is the same in every OOM error