Performance bottleneck: consider multiprocessing for cached request checking

[justHungryMan]

Currently, the following loop runs in a single process, which can be a bottleneck when checking a large number of cached requests:

lm-evaluation-harness/lm_eval/api/model.py

Lines 264 to 284 in a96085f

	for req in tqdm(requests, desc="Checking cached requests"):
	hsh = hash_args(attr, req.args)
	if attr == "generate_until" and req.args[1].get("do_sample", False):
	# when we are doing non-greedy generation, don't use the cache
	# (else every "randomly sampled" generation would be identical for repeats > 1).
	if not warned:
	eval_logger.warning(
	f"Arguments to lm.generate_until() '{req.args[1]}' include non-deterministic sampling. Caching will not be performed for such requests."
	)
	warned = True
	res.append(None)
	remaining_reqs.append(req)
	elif hsh in self.dbdict:
	ob = self.dbdict[hsh]
	assert ob is not None
	res.append(ob)
	else:
	res.append(None)
	remaining_reqs.append(req)

Since this part of the code is only computing hashes and checking for existence in self.dbdict, it seems like a good candidate for parallelization using multiprocessing.

Would it be possible to explore using Python’s multiprocessing module or similar (e.g., concurrent.futures.ProcessPoolExecutor) to speed up this step?

This could significantly reduce the latency during evaluation or repeated runs, especially when dealing with a large number of requests.

Let me know if this is something you’d be open to – I’d be happy to explore or contribute a PR if helpful.