Generation length is limited to 2048 tokens.Qwen3 model accuracy is low

[sravan500]

Hi Team,
I am using local-completions to evaluate my model deployed in local.
The generated response is limited to 2048 tokens,and not able to increase it throught model_args or gen_kwargs max_length=120000 also.
Did anyone face this issue?

[KMouratidis]

Not with the bf16 or my own AWQ quants, but with Qwen's 32B-AWQ model, it seems I'm hitting some weird behavior:

~$ lm_eval --model local-completions --tasks gsm8k --model_args model=Qwen/Qwen3-32B-AWQ,base_url=http://127.0.0.1:11435/v1/completions,max_retries=5,timeout=1200,max_length=32768,num_concurrent=200 --num_fewshot 5  --limit 10 --log_samples --output_path=./results
INFO 05-03 17:04:42 [importing.py:53] Triton module has been replaced with a placeholder.
INFO 05-03 17:04:42 [__init__.py:239] Automatically detected platform cuda.
2025-05-03:17:04:48,528 WARNING  [lm_eval.__main__:316]  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2025-05-03:17:04:48,528 INFO     [lm_eval.__main__:379] Selected Tasks: ['gsm8k']
2025-05-03:17:04:48,529 INFO     [lm_eval.evaluator:169] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-05-03:17:04:48,529 INFO     [lm_eval.evaluator:206] Initializing local-completions model, with arguments: {'model': 'Qwen/Qwen3-32B-AWQ', 'base_url': 'http://127.0.0.1:11435/v1/completions', 'max_retries': 5, 'timeout': 1200, 'max_length': 32768, 'num_concurrent': 200}
2025-05-03:17:04:48,529 INFO     [lm_eval.models.api_models:115] Using max length 32768 - 1
2025-05-03:17:04:48,529 INFO     [lm_eval.models.api_models:133] Using tokenizer huggingface
2025-05-03:17:04:54,037 WARNING  [lm_eval.evaluator:275] Overwriting default num_fewshot of gsm8k from 5 to 5
2025-05-03:17:04:54,038 INFO     [lm_eval.api.task:420] Building contexts for gsm8k on rank 0...
100%|█████████| 10/10 [00:00<00:00, 535.92it/s]
2025-05-03:17:04:54,057 INFO     [lm_eval.evaluator:517] Running generate_until requests
Requesting API: 100%|█████████| 10/10 [00:06<00:00,  1.55it/s]
fatal: not a git repository (or any of the parent directories): .git
2025-05-03:17:05:01,082 INFO     [lm_eval.loggers.evaluation_tracker:209] Saving results aggregated
2025-05-03:17:05:01,083 INFO     [lm_eval.loggers.evaluation_tracker:290] Saving per-sample results for: gsm8k
local-completions (model=Qwen/Qwen3-32B-AWQ,base_url=http://127.0.0.1:11435/v1/completions,max_retries=5,timeout=1200,max_length=32768,num_concurrent=200), gen_kwargs: (None), limit: 10.0, num_fewshot: 5, batch_size: 1
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |    0|±  |     0|
|     |       |strict-match    |     5|exact_match|↑  |    0|±  |     0|

[InfroLab]

With qwen/qwen3-32b from OpenRouter I get these messages spammed in console:

2025-05-27:00:24:32 WARNING [models.api_models:515] Context length (2058) + continuation length (1) > max_length (2047). Left truncating context.

and then it crashes with following stacktrace:
Requesting API: 0%| | 0/549473 [00:00<?, ?it/s]Traceback (most recent call last): File "/home/InfroLab/bench310/bin/lm_eval", line 8, in <module> sys.exit(cli_evaluate()) File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/__main__.py", line 449, in cli_evaluate results = evaluator.simple_evaluate( File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/utils.py", line 439, in _wrapper return fn(*args, **kwargs) File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/evaluator.py", line 338, in simple_evaluate results = evaluate( File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/utils.py", line 439, in _wrapper return fn(*args, **kwargs) File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/evaluator.py", line 570, in evaluate resps = getattr(lm, reqtype)(cloned_reqs) File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/api/model.py", line 382, in loglikelihood return self._loglikelihood_tokens(new_reqs, disable_tqdm=disable_tqdm) File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/models/api_models.py", line 621, in _loglikelihood_tokens asyncio.run( File "/home/InfroLab/bench310/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/home/InfroLab/bench310/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/models/api_models.py", line 565, in get_batched_requests return await tqdm_asyncio.gather(*tasks, desc="Requesting API") File "/home/InfroLab/bench310/lib/python3.10/site-packages/tqdm/asyncio.py", line 79, in gather res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout, File "/home/InfroLab/bench310/lib/python3.10/site-packages/tqdm/asyncio.py", line 79, in <listcomp> res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout, File "/home/InfroLab/bench310/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one return f.result() # May raise f.exception(). File "/home/InfroLab/bench310/lib/python3.10/site-packages/tqdm/asyncio.py", line 76, in wrap_awaitable return i, await f File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped return await copy(fn, *args, **kwargs) File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__ do = await self.iter(retry_state=retry_state) File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 153, in iter result = await action(retry_state) File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/_utils.py", line 99, in inner return call(*args, **kwargs) File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/__init__.py", line 420, in exc_check raise retry_exc.reraise() File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/__init__.py", line 187, in reraise raise self.last_attempt.result() File "/home/InfroLab/bench310/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.__get_result() File "/home/InfroLab/bench310/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/home/InfroLab/bench310/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__ result = await fn(*args, **kwargs) File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/models/api_models.py", line 487, in amodel_call else self.parse_logprobs( File "/home/InfroLab/bench310/lib/python3.10/site-packages/lm_eval/models/openai_completions.py", line 76, in parse_logprobs sorted(out["choices"], key=itemgetter("index")), ctxlens KeyError: 'choices' Requesting API: 0%| | 0/549473 [11:32<?, ?it/s

Here is a launch command:
lm_eval --model local-completions --tasks m_mmlu_ru,m_mmlu_es,m_mmlu_en,m_mmlu_fr,m_mmlu_hi,hellaswag_ru,hellaswag_es,hellaswag,hellaswag_fr,hellaswag_hi,arc_ru,ai2_arc,arc_es,arc_fr,arc_hi,truthfulqa,truthfulqa_es_mc1,truthfulqa_fr_mc1,truthfulqa_hi_mc1,truthfulqa_ru_mc1,truthfulqa_es_mc2,truthfulqa_fr_mc2,truthfulqa_hi_mc2,truthfulqa_ru_mc2,belebele_rus_Cyrl,belebele_eng_Latn,belebele_hin_Deva,belebele_spa_Latn,belebele_fra_Latn --model_args model=qwen/qwen3-32b,base_url=https://openrouter.ai/api/v1/completions,num_concurrent=8,max_retries=3,tokenized_requests=False

lm-eval installed from git with a commit 8bc4aff

[InfroLab]

Also, another model qwen/qwen3-30b-a3b shows these error during same lm-eval command benchmark:
{"error":{"message":"Provider returned error","code":400,"me tadata":{"raw":"{\"error\":{\"object\":\"error\",\"type\":\"invalid_request_error\",\"message\":\"2 request validation errors : Missing required input field: 'prompt'; Extra inputs are not permitted, field: 'messages', value: [{'role': 'user', 'conten t': '1 मार्च को, एक आदमी और एक ठेकेदार एक समझौते पर हस्ताक्षर किए जिसमें यह उल्लेख किया गया था कि ठेकेदार हर आठ निर्दिष्ट समुद्र तट घर में एक दो-स्तरीय रेडवुड डेक निर् नाम से होंगे, जिसमें सभी काम 1 मई तक पूरे होंगे। संविदा निम्नलिखित प्रदान करता है:\\\"हर दोहरी टेरेस पर लागत $2,500 है, प्रत्येक आठ समुद्र तट घर मेंतान किया जाना चाहिए।\\\" 25 मार्च की सुबह, जब ठेकेदार और उसके सहायक चौथे टेरेस पर काम पूरा कर रहे थे, तब आदमी एक मुकाम पर गया क्योंकि उन्होंने अ और दशमलव के अनुरूप उसके पास समान नकदी वाली एक ताकत वाली समुद्र तट घरों में चार में से एक पर टेरेस निर्माण करना चाहते हैं। अगर आप उस अंतिम महत्य तक पहुंच सकते हैं, तो मैं आपको उस समय $10,000 वापस करूंगा।\\\" ठेकेदार ने उत्तर दिया, \\\"सहमत हुए। हम हाथ [96/1857$ वपूर्ण समयासे पहलेहउसमेंसमयितकसपहुंच सकतेघहैंों तो मैंाआपकोाउसरसमयया था और वापसअकरूंगा।र घरोंठेकेदारसनेकेउत्तरमदियाा काम सहमतरहुए।यहमथहाथ2मिलाते हैं।\\\" और हाथ मि में आठों में से चार को पूरी तरह से नष्ट कर दिया। इसके अतिरिक्त, तूफान ने ठेकेदार द्वारा निर्मित टाउन बीच हाउस पर 85 फीसदी काम पूरक आदमीसकेासमुद्रभतट घरोंसेमेंघसेयचारोकोएपूरीऔतरहनसेीनष्टीकरपदिया।कइसकेहअतिरिक्त। तूफानआनेमठेकेदारठेद्वारासेनिर्मितिटाउनजबीच हाउस पर कहनफीसदी\कामतूपूरकरने के बाद तोड़ दिया। दो ठेकेदाो गएएथेदऔरदंउनकी सभी उपकरणोंाको बहशेगयाीथा।मुफिरतआदमीोंकोरठेकेदारपूसेापत्रींमिलासकजिसमेंकृउनकामुयहशकहना था किए तूफामेरी उपकरणों को तबाह कर घर है। मैंरेएकदमरदंगबरहएगया हूँरेऔरकेताउन में2शेष0तीनजेंसमुद्र"तटफघरोंसेपरहकाम पूरामनहींकेकरिसकता। कृपया मुझेशशहरयमें मेरेजकिएठेगए काम के लिए $20,000 हैसमयितकसपहुंच सप्रतिशततपूरेमैंकामककेालिएसबकायाथराशि केासाथकरूंगा।र घरोंठेकेदारसनेकेउत्तरमदियाा काम सहमतप्रतिशत:पूरे:काम[के लिए बकाया राशि के साथ घरों में से चार को पूरी तरह से नष्ट क━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━निर्मित टाउन बीच हाउस प

I tried test requests to these API using openai-python package and they return all the necessary data.