[Usage]: Qwen/Qwen2-VL-7B-Instruct #10994

mahmoudelnazer · 2024-12-08T19:46:28Z

The model to consider.

https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct

The closest model vllm already supports.

No response

What's your difficulty of supporting the model you want?

i deployed the model using vllm using the below command .
vllm serve Qwen/Qwen2-VL-7B-Instruct

How could i call this endpoint using curl command ?

i used this curl command:
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"Qwen/Qwen2-VL-7B-Instruct","messages":[{"role":"system","content":"You are Qwen, a vision-language AI assistant created by Alibaba Cloud."},{"role":"user","content":"Describe this image in detail. [Image: '"$(base64 -w 0 /home/azureuser/qwen/test.jpeg)"']"}],"temperature":0.7,"top_p":0.8,"max_tokens":100}'

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

DarkLight1337 · 2024-12-08T22:31:12Z

Please read the examples on this page.

mahmoudelnazer · 2024-12-09T10:19:03Z

thanks @DarkLight1337 the link shows the python configuration of Phi-3.5-vision-instruct not Qwen 2 vL instruct.
also itdid not show how could i pass the images using the curl command.

DarkLight1337 · 2024-12-09T13:44:05Z

Our online server follows OpenAI API spec. You can refer to their official docs on some example curl commands.

mahmoudelnazer · 2024-12-10T12:06:38Z

thanks @DarkLight1337 this documentation shows me to use images , i want to use video also . could you send me any documentation of it ?

DarkLight1337 · 2024-12-10T12:09:45Z

You can replace image_url with video_url and send a base64 video URL.

DarkLight1337 · 2024-12-10T12:11:40Z

Please read the examples on this page.

There is also a short section on video input here.

yumlevi · 2024-12-25T05:34:00Z

Hi, did you get this sorted by any chance? trying to load a base64 video and it doesn't seem to work at all

DarkLight1337 · 2024-12-25T06:33:17Z

Can you show how you're sending the video to the model?

DarkLight1337 · 2024-12-25T06:59:07Z

Maybe it is related to this bug I found: huggingface/transformers#35412

yumlevi · 2024-12-25T10:38:21Z

all I'm doing is opening a video file (that works) converting it to base64 and sending it along, actually doing a URL works by the way, it's just the base64 that seems broken.

import requests
import json
import random
import base64
import os
file_loc = '/path/to/video/test2.mp4'
#open file
with open(file_loc, 'rb') as file:
    video_data = file.read()
    video_base64 = base64.b64encode(video_data).decode('utf-8')
url = "http://localhost:6001/v1/chat/completions"
headers = {
    "Content-Type": "application/json"
}
data = {
    "model": "Qwen/Qwen2-VL-7B-Instruct",
    "messages": [{
            "role": "user",
            "content": [
            {
                "type": "video_url",
                "video_url": {
                    "url": f"data:video/jpeg;base64,{video_base64}"
                }
            },
            {
                "type": "text",
                "text": "make a short description for this video"
            },
        ],
        }
    ]
}

response = requests.post(url, headers=headers, json=data)
print(response.text)

produces this error

File ".venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File ".venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File ".venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File ".venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 185, in __call__
    with collapse_excgroups():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File ".venv/lib/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File ".venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 187, in __call__
    response = await self.dispatch_func(request, call_next)
  File ".venv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 543, in add_request_id
    response = await call_next(request)
  File ".venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 163, in call_next
    raise app_exc
  File ".venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 149, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File ".venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File ".venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File ".venv/lib/python3.10/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File ".venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File ".venv/lib/python3.10/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
  File ".venv/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
  File ".venv/lib/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
  File ".venv/lib/python3.10/site-packages/vllm/entrypoints/utils.py", line 54, in wrapper
    return handler_task.result()
  File ".venv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 367, in create_chat_completion
    generator = await handler.create_chat_completion(request, raw_request)
  File ".venv/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 161, in create_chat_completion
    ) = await self._preprocess_chat(
  File ".venv/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_engine.py", line 485, in _preprocess_chat
    mm_data = await mm_data_future
  File ".venv/lib/python3.10/site-packages/vllm/entrypoints/chat_utils.py", line 492, in all_mm_data
    items = await asyncio.gather(*self._items)
  File ".venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 294, in async_get_and_parse_video
    video = await async_fetch_video(video_url)
  File ".venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 200, in async_fetch_video
    video = _load_video_from_data_url(video_url)
  File ".venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 161, in _load_video_from_data_url
    return np.stack([
  File ".venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 162, in <listcomp>
    load_video_frames_from_base64(frame_base64)
  File ".venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 135, in load_video_frames_from_base64
    return _load_video_frames_from_bytes(base64.b64decode(frame))
  File ".venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 129, in _load_video_frames_from_bytes
    frame = Image.open(BytesIO(b))
  File ".venv/lib/python3.10/site-packages/PIL/Image.py", line 3536, in open
    """
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x14b0ee37bab0>

DarkLight1337 · 2024-12-25T12:24:10Z

It looks like you're sending a MP4 video. In that case, the media type of the URL should be video/mp4, not video/jpeg.

The available media types are listed here: https://www.iana.org/assignments/media-types/media-types.xhtml

yumlevi · 2024-12-25T13:21:10Z

thanks for trying to help, changed to "url": f"data:video/mp4;base64,{video_base64}", unfortunately get this error

  load_video_frames_from_base64(frame_base64)
  File "/.venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 135, in load_video_frames_from_base64
    return _load_video_frames_from_bytes(base64.b64decode(frame))
  File "/.venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 129, in _load_video_frames_from_bytes
    frame = Image.open(BytesIO(b))
  File "/.venv/lib/python3.10/site-packages/PIL/Image.py", line 3498, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x14b213101fd0>

DarkLight1337 · 2024-12-25T13:40:08Z

thanks for trying to help, changed to "url": f"data:video/mp4;base64,{video_base64}", unfortunately get this error

  load_video_frames_from_base64(frame_base64)
  File "/.venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 135, in load_video_frames_from_base64
    return _load_video_frames_from_bytes(base64.b64decode(frame))
  File "/.venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 129, in _load_video_frames_from_bytes
    frame = Image.open(BytesIO(b))
  File "/.venv/lib/python3.10/site-packages/PIL/Image.py", line 3498, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x14b213101fd0>

Can you try out #11492? It should fix your problem.

yumlevi · 2024-12-25T14:26:40Z

yep the commit works.

mahmoudelnazer · 2024-12-26T11:05:57Z

yep the commit
could you type the curl that works on giving the model a video ?

DarkLight1337 · 2024-12-26T11:11:14Z

all I'm doing is opening a video file (that works) converting it to base64 and sending it along, actually doing a URL works by the way, it's just the base64 that seems broken.

import requests
import json
import random
import base64
import os
file_loc = '/path/to/video/test2.mp4'
#open file
with open(file_loc, 'rb') as file:
    video_data = file.read()
    video_base64 = base64.b64encode(video_data).decode('utf-8')
url = "http://localhost:6001/v1/chat/completions"
headers = {
    "Content-Type": "application/json"
}
data = {
    "model": "Qwen/Qwen2-VL-7B-Instruct",
    "messages": [{
            "role": "user",
            "content": [
            {
                "type": "video_url",
                "video_url": {
                    "url": f"data:video/jpeg;base64,{video_base64}"
                }
            },
            {
                "type": "text",
                "text": "make a short description for this video"
            },
        ],
        }
    ]
}

response = requests.post(url, headers=headers, json=data)
print(response.text)

produces this error

File ".venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File ".venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File ".venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File ".venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 185, in __call__
    with collapse_excgroups():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File ".venv/lib/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File ".venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 187, in __call__
    response = await self.dispatch_func(request, call_next)
  File ".venv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 543, in add_request_id
    response = await call_next(request)
  File ".venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 163, in call_next
    raise app_exc
  File ".venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 149, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File ".venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File ".venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File ".venv/lib/python3.10/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File ".venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    raise exc
  File ".venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    await app(scope, receive, sender)
  File ".venv/lib/python3.10/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
  File ".venv/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
  File ".venv/lib/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
  File ".venv/lib/python3.10/site-packages/vllm/entrypoints/utils.py", line 54, in wrapper
    return handler_task.result()
  File ".venv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 367, in create_chat_completion
    generator = await handler.create_chat_completion(request, raw_request)
  File ".venv/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 161, in create_chat_completion
    ) = await self._preprocess_chat(
  File ".venv/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_engine.py", line 485, in _preprocess_chat
    mm_data = await mm_data_future
  File ".venv/lib/python3.10/site-packages/vllm/entrypoints/chat_utils.py", line 492, in all_mm_data
    items = await asyncio.gather(*self._items)
  File ".venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 294, in async_get_and_parse_video
    video = await async_fetch_video(video_url)
  File ".venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 200, in async_fetch_video
    video = _load_video_from_data_url(video_url)
  File ".venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 161, in _load_video_from_data_url
    return np.stack([
  File ".venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 162, in <listcomp>
    load_video_frames_from_base64(frame_base64)
  File ".venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 135, in load_video_frames_from_base64
    return _load_video_frames_from_bytes(base64.b64decode(frame))
  File ".venv/lib/python3.10/site-packages/vllm/multimodal/utils.py", line 129, in _load_video_frames_from_bytes
    frame = Image.open(BytesIO(b))
  File ".venv/lib/python3.10/site-packages/PIL/Image.py", line 3536, in open
    """
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x14b0ee37bab0>

You can refer to this one.

mahmoudelnazer added the new model Requests to new models label Dec 8, 2024

DarkLight1337 added usage How to use vllm and removed new model Requests to new models labels Dec 8, 2024

DarkLight1337 changed the title ~~[New Model]: Qwen/Qwen2-VL-7B-Instruct~~ [Usage]: Qwen/Qwen2-VL-7B-Instruct Dec 8, 2024

DarkLight1337 mentioned this issue Dec 25, 2024

[Frontend] Enable decord to load video from base64 #11492

Merged

DarkLight1337 closed this as completed in #11492 Dec 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Usage]: Qwen/Qwen2-VL-7B-Instruct #10994

[Usage]: Qwen/Qwen2-VL-7B-Instruct #10994

mahmoudelnazer commented Dec 8, 2024

DarkLight1337 commented Dec 8, 2024

mahmoudelnazer commented Dec 9, 2024

DarkLight1337 commented Dec 9, 2024

mahmoudelnazer commented Dec 10, 2024

DarkLight1337 commented Dec 10, 2024

DarkLight1337 commented Dec 10, 2024

yumlevi commented Dec 25, 2024

DarkLight1337 commented Dec 25, 2024

DarkLight1337 commented Dec 25, 2024

yumlevi commented Dec 25, 2024 •

edited

Loading

DarkLight1337 commented Dec 25, 2024

yumlevi commented Dec 25, 2024 •

edited

Loading

DarkLight1337 commented Dec 25, 2024

yumlevi commented Dec 25, 2024

mahmoudelnazer commented Dec 26, 2024

DarkLight1337 commented Dec 26, 2024

[Usage]: Qwen/Qwen2-VL-7B-Instruct #10994

[Usage]: Qwen/Qwen2-VL-7B-Instruct #10994

Comments

mahmoudelnazer commented Dec 8, 2024

The model to consider.

The closest model vllm already supports.

What's your difficulty of supporting the model you want?

Before submitting a new issue...

DarkLight1337 commented Dec 8, 2024

mahmoudelnazer commented Dec 9, 2024

DarkLight1337 commented Dec 9, 2024

mahmoudelnazer commented Dec 10, 2024

DarkLight1337 commented Dec 10, 2024

DarkLight1337 commented Dec 10, 2024

yumlevi commented Dec 25, 2024

DarkLight1337 commented Dec 25, 2024

DarkLight1337 commented Dec 25, 2024

yumlevi commented Dec 25, 2024 • edited Loading

DarkLight1337 commented Dec 25, 2024

yumlevi commented Dec 25, 2024 • edited Loading

DarkLight1337 commented Dec 25, 2024

yumlevi commented Dec 25, 2024

mahmoudelnazer commented Dec 26, 2024

DarkLight1337 commented Dec 26, 2024

yumlevi commented Dec 25, 2024 •

edited

Loading

yumlevi commented Dec 25, 2024 •

edited

Loading