Swallowed Exception Causes Unhelpful Error Message When Using structured_outputs and response_model

[husniadil]

Summary

When using structured_outputs and response_model, if an error occurs—such as a network error or an incorrect API key—the agent incorrectly returns an unhelpful error message. For instance:

You tried to pass a `BaseModel` class to `chat.completions.create()`. You must use `beta.chat.completions.parse()`.

Expected Behavior

The system should display the actual error message, such as:

Connection error.

or

Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-proj-*********************************************************************************************************************************************************XM8A. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

The issue arises from a swallowed exception in the following code section:

phidata/phi/model/openai/chat.py

Lines 340 to 342 in 48addb4

	except Exception as e:
	logger.error(f"Error from OpenAI API: {e}")

Is this behavior intentional, or is it a bug that needs to be addressed?

I exclusively use OpenAI, so I do not explore other models.

[manthanguptaa]

@husniadil can you help me with your code to replicate this?

[husniadil]

@manthanguptaa Please kindly take a look:

from typing import Dict
from phi.agent import Agent
from phi.model.openai import OpenAILike
from typing import Optional
from phi.tools import Toolkit
from pydantic import BaseModel, Field


class Weather(BaseModel):
    temperature: Optional[float]
    humidity: Optional[float]
    wind_speed: Optional[float]
    wind_direction: Optional[str]
    weather: Optional[str]


class Response(BaseModel):
    weather: Weather | None = Field(None, description="Weather data for the city.")
    error: str | None = Field(None, description="Error message.")


class WeatherToolkit(Toolkit):
    """
    Toolkit for getting weather data.
    """

    def __init__(self):
        super().__init__(name="weather_toolkit")
        self.register(self.get_by_city)

    weather_data: Dict[str, Weather] = {
        "New York": Weather(
            temperature=70,
            humidity=0.5,
            wind_speed=10,
            wind_direction="NE",
            weather="Sunny",
        ),
        "Los Angeles": Weather(
            temperature=80,
            humidity=0.3,
            wind_speed=5,
            wind_direction="NW",
            weather="Cloudy",
        ),
        "Chicago": Weather(
            temperature=60,
            humidity=0.7,
            wind_speed=15,
            wind_direction="SE",
            weather="Rainy",
        ),
        "Houston": Weather(
            temperature=75,
            humidity=0.4,
            wind_speed=8,
            wind_direction="SW",
            weather="Sunny",
        ),
    }

    def get_by_city(self, city: str) -> str | None:
        """
        Get weather data for a city.

        Args:
            city: The name of the city.

        Returns:
            Weather: The weather data for the city.
        """
        return str(self.weather_data.get(city))


weather_agent = Agent(
    name="Weather Agent",
    model=OpenAILike(
        api_key="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",
        id="gpt-4o-mini",
    ),
    tools=[WeatherToolkit()],
    instructions=[
        """
        You have access to weather data for New York, Los Angeles, Chicago, and Houston.
        If the user asks for the weather in one of these cities, you should return the weather data.
        If the user asks for the weather in a city that is not in the list, you should tell the user that the city is not in the list.
        """,
    ],
    structured_outputs=True,
    parse_response=True,
    response_model=Response,
)

if __name__ == "__main__":
    import sys

    args = sys.argv[1:]
    city = args[0] if args else None

    if not city:
        print("Please provide a city name.")
        sys.exit(1)

    try:
        response = weather_agent.run(f"What is the weather in {city}?")
        output = Response.model_validate(response.content)
        print(output.model_dump_json())
    except Exception as e:
        output = Response(error=str(e))
        print(output.model_dump_json())

Run with a valid API key

> python main.py
Please provide a city name.

> python main.py "New York"
{"weather":{"temperature":70.0,"humidity":0.5,"wind_speed":10.0,"wind_direction":"NE","weather":"Sunny"},"error":null}

> python main.py Solo
{"weather":null,"error":"The city is not in the list."}

Run with a invalid API key (swallow exception)

> python main.py Chicago
ERROR    Error from OpenAI API: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-proj-**3456. You can find your API key at https://platform.openai.com/account/api-keys.', 'type':
         'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}
{"weather":null,"error":"You tried to pass a `BaseModel` class to `chat.completions.create()`; You must use `beta.chat.completions.parse()` instead"}

Run with a invalid API key (raise exception)

> python main.py Houston
ERROR    Error from OpenAI API: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-proj-**3456. You can find your API key at https://platform.openai.com/account/api-keys.', 'type':
         'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}
{"weather":null,"error":"Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-proj-**3456. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}"}

Those are the examples when we run it with incorrect API key.

If I disconnected my internet and run it, I got:

> python main.py "Los Angeles"
ERROR    Error from OpenAI API: Connection error.
{"weather":null,"error":"You tried to pass a `BaseModel` class to `chat.completions.create()`; You must use `beta.chat.completions.parse()` instead"}

Notice that the JSON response is not straightforward.

Let me know if I am using this wrongly.

[manthanguptaa]

So @husniadil we bubble up the error from the model and don't send it on our own. I will test your code nevertheless and get back to you.