[Usage]: use vllm to serve gguf model with cpu only

[pamdla]

Your current environment

I was trying to use vllm like this:

vllm serve /opt/llm/dpsk/DeepSeek-R1-Distill-Qwen-1.5B-Q8_0.gguf --quantization gguf --device cpu --tokenizer ./DeepSeek-R1-Distill-Qwen-1.5B

and got this error:

  File "/opt/miniconda3/envs/locallm/lib/python3.12/site-packages/vllm/platforms/interface.py", line 165, in is_async_output_supported
    raise NotImplementedError
NotImplementedError

Don't know if it is possible to use vllm to serve gguf model with CPU only.

How would you like to use vllm

I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.
need help. thanks.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.