[broken-compatibility] chat completion breaks base64 standard / openAI spec

[lucyknada]

System Info

latest docker pull, --version says: text-generation-launcher 3.0.0
model used: https://huggingface.co/AI-Safeguard/Ivy-VL-llava

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

1. host any vision model with tgi
2. use any openAI / chat-completions compatible frontend or library, e.g. openwebUI to connect to it on /v1
3. send an image and user-message
4. it gets auto-converted into base64 and put into context as image-url (openAI spec)
5. the response clearly indicates it sees the base64, but the model simply makes fun of that fact instead of tgi injecting it as image embeddings

---

model response:

payload sent by frontends and openAI-spec:

meanwhile tgi expects I assume base64 in an "inputs" key separate from messages or ignores base64 in messages entirely?

either way that breaks compatibility with all frontends and libraries that expect the backend to handle in-line base64.

thanks!

referencing original issue for cross-repo visibility: open-webui/open-webui#7760

Expected behavior

base64 image embedded into context is recognized as is with no markdown urls