Support int8 kvcahe

[sleepcoo]

Motivation

Implemented online int8 asymmetric quantization for kv cache.

In Summary

the current int8 kvcache has the following advantages:

• Online kv cache int8 quantization does not require an offline dataset.
• KV int8 quantization has almost lossless accuracy.
• In scenarios with high computational power and low GPU memory, there are significant benefits.

I also tested lmdeploy, and its turbomind showed performance gains, but the gains with the PyTorch engine are also minimal and even negative in some scenarios due to the overhead of attn dequant. I will later investigate the implementation of lmdeploy turbomind to see how it can be integrated into sglang, and support Online kv cache quantization with fp8 as well. cc @zhyncs

Quantitative Method

We adopted int8 asymmetric quantization to quantize the kv cache online using per token and per head methods.

Usage

Using KV quantization and reasoning through sglang is very simple. Only need to set the parameter to --kv-cache-dtype=int8. ( currently only the Triton backend is supported, the Flashinfer backend is coming soon.@yinfan98 )

Evaluation

We used the benchmark methods benchmark/gsm8k and benchmark/mmlu in sglang. The results are shown in the table below:

model	dataset	metric	Int8	Fp16
Qwen2.5 14B	gsm8k	accuracy	0.718	0.723
LLama2-13B	gsm8k	accuracy	0.234	0.234
LLama3-8B	gsm8k	accuracy	0.752	0.752

Test command

python3 benchmark/gsm8k/bench_sglang.py  --num-questions 1319

Performance

test on A100-40G

model	kv type	RPS	v.s. kv fp16
Qwen2.5 14B	fp16	10.48	1
Qwen2.5 14B	int8	10.41	0.996
Llama2 13B	fp16	5.21	1
llama2 13B	int8	5.97	1.15

todo

• Add unit tests
• Add evaluation results

[yinfan98]

Please help review it again. cc: @ispobock @zhyncs @sleepcoo

[ispobock]

Why using eagle target model for test？You can specify the model name here or add a new constant for MHA model.