Error occurs when loading the gemma model in bitsandbytes format.

[upskyy]

Motivation

#2556

Modifications

I refer to vllm to solve the key error that occurs when loading in bitsandbytes format.

An error occurs when loading the gemma model with the command below.

python3 -m sglang.launch_server --model-path /models --tokenizer-path /models --port 30000 --tokenizer-mode auto --dtype bfloat16 --mem-fraction-static 0.5 --random-seed 0 --enable-torch-compile --disable-cuda-graph --schedule-conservativeness 1.3 --kv-cache-dtype fp8_e5m2 --load-format bitsandbytes --quantization bitsandbytes

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

[merrymercy]

@upskyy Thanks for the contribution. It is merged. Can you also help us fix this llama model? #2600

[upskyy]

@merrymercy
Looking at the code, I think the issue will also be resolved with the PR #2557. This problem occurred when changing to qweight when doing bitsandbytes 4bit load, but in the PR #2557, it was corrected not only for the gemma model but also for loads of other models.