[Bug]: can not serve microsoft/llava-med-v1.5-mistral-7b

[cubense]

Your current environment

The output of `python collect_env.py`

Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Rocky Linux release 8.10 (Green Obsidian) (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-22)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.28

Python version: 3.10.15 | packaged by conda-forge | (main, Oct 16 2024, 01:24:24) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-4.18.0-553.16.1.el8_10.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA H100 NVL
GPU 1: NVIDIA H100 NVL

Nvidia driver version: 550.54.15
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              64
On-line CPU(s) list: 0-63
Thread(s) per core:  1
Core(s) per socket:  32
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               106
Model name:          Intel(R) Xeon(R) Platinum 8358P CPU @ 2.60GHz
Stepping:            6
CPU MHz:             3400.000
BogoMIPS:            5200.00
Virtualization:      VT-x
L1d cache:           48K
L1i cache:           32K
L2 cache:            1280K
L3 cache:            49152K
NUMA node0 CPU(s):   0-31
NUMA node1 CPU(s):   32-63
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoinvd dtherm ida arat pln pts hwp_epp avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.2.0
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.46.2
[pip3] triton==3.0.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.1.3.1                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.1.105                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.0.2.54                pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.2.106               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.4.5.107               pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.1.0.106               pypi_0    pypi
[conda] nvidia-ml-py              12.560.30                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.1.105                 pypi_0    pypi
[conda] pyzmq                     26.2.0                   pypi_0    pypi
[conda] torch                     2.4.0                    pypi_0    pypi
[conda] torchvision               0.19.0                   pypi_0    pypi
[conda] transformers              4.46.2                   pypi_0    pypi
[conda] triton                    3.0.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.3.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    NIC0    NIC1    NIC2    NIC3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     SYS     SYS     SYS     SYS     0-31    0               N/A
GPU1    SYS      X      SYS     SYS     SYS     SYS     0-31    0               N/A
NIC0    SYS     SYS      X      PIX     SYS     SYS
NIC1    SYS     SYS     PIX      X      SYS     SYS
NIC2    SYS     SYS     SYS     SYS      X      PIX
NIC3    SYS     SYS     SYS     SYS     PIX      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3

Model Input Dumps

No response

🐛 Describe the bug

I can successfully use
python -m vllm.entrypoints.openai.api_server --model llava-hf/llava-1.5-7b-hf --chat-template /condo/wanglab/shared/xyw/vllm/examples/template_llava.jinja
, but can not use 'microsoft/llava-med-v1.5-mistral-7b', which is supported by vLLM
and I can not use this model just using vllm serve "microsoft/llava-med-v1.5-mistral-7b"
this model can be loaded in transformer:
from llava.model.builder import load_pretrained_model model_path='microsoft/llava-med-v1.5-mistral-7b' model_base=None model_name='llava-med-v1.5-mistral-7b' tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, model_base, model_name, load_8bit=False, load_4bit=False, device="cuda")

I don't know how to use it in vllm

python -m vllm.entrypoints.openai.api_server --model microsoft/llava-med-v1.5-mistral-7b --chat-template /condo/wanglab/shared/xyw/vllm/examples/template_llava.jinja
INFO 11-07 14:36:42 api_server.py:528] vLLM API server version 0.6.3.post1
INFO 11-07 14:36:42 api_server.py:529] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template='/condo/wanglab/shared/xyw/vllm/examples/template_llava.jinja', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='microsoft/llava-med-v1.5-mistral-7b', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False)
INFO 11-07 14:36:42 api_server.py:166] Multiprocessing frontend to use ipc:///tmp/53b633e0-ca5c-4528-9119-924df1cab8e9 for IPC Path.
INFO 11-07 14:36:42 api_server.py:179] Started engine process with PID 856389
Traceback (most recent call last):
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1034, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 736, in getitem
raise KeyError(key)
KeyError: 'llava_mistral'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 585, in
uvloop.run(run_server(args))
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/uvloop/init.py", line 82, in run
return loop.run_until_complete(wrapper())
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/uvloop/init.py", line 61, in wrapper
return await main
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 552, in run_server
async with build_async_engine_client(args) as engine_client:
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 107, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 184, in build_async_engine_client_from_engine_args
engine_config = engine_args.create_engine_config()
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 903, in create_engine_config
model_config = self.create_model_config()
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 839, in create_model_config
return ModelConfig(
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/config.py", line 162, in init
self.hf_config = get_config(self.model, trust_remote_code, revision,
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/transformers_utils/config.py", line 202, in get_config
raise e
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/transformers_utils/config.py", line 183, in get_config
config = AutoConfig.from_pretrained(
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1036, in from_pretrained
raise ValueError(
ValueError: The checkpoint you are trying to load has model type llava_mistral but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
Process SpawnProcess-1:
Traceback (most recent call last):
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1034, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 736, in getitem
raise KeyError(key)
KeyError: 'llava_mistral'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 390, in run_mp_engine
engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 135, in from_engine_args
engine_config = engine_args.create_engine_config()
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 903, in create_engine_config
model_config = self.create_model_config()
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 839, in create_model_config
return ModelConfig(
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/config.py", line 162, in init
self.hf_config = get_config(self.model, trust_remote_code, revision,
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/transformers_utils/config.py", line 202, in get_config
raise e
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/vllm/transformers_utils/config.py", line 183, in get_config
config = AutoConfig.from_pretrained(
File "/condo/wanglab/tmhyxx23/conda/llava/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1036, in from_pretrained
raise ValueError(
ValueError: The checkpoint you are trying to load has model type llava_mistral but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

The model seems to be based on LLaVA GitHub repo rather than in HF format. Please see #7984 (comment)

[DarkLight1337]

Refer to discussion on #11449