Required parameters for function calling - asking user for missing information

[deepinderdeol]

User should be prompted for required parameters that are not specified in the user request. However, we have noticed inconsistent behavior when all parameters are not provided in the user request and many times the model ends up providing missing information on its own.

We have tried giving system instructions, providing instructions in function/parameter descriptions etc. but we haven't been able to get consistent behavior. What is the best way to handle this?

Filters can be used, but that won't work for all scenarios because the parameters chosen by the model could be valid - but the information should ideally come from the user.

[evchaki]

@deepinderdeol - which model are you using for this?

[deepinderdeol]

@evchaki We have tried GPT-4o (5/13 version) and other older GPT-4 models.

Recently I have also tried many of the open-source models with the Ollama connector - llama3.2, qwen2 etc. - but maybe these handle function calling differently...

[evchaki]

@markwallace-microsoft - FYI

[markwallace-microsoft]

@deepinderdeol
One thing you can try is:

1. Don't advertise the parameters to the LLM is the user is supposed to provide them. Take a look at this sample TransformPlugin.cs which is not exactly what you need but does should how to remove parameters from a function.
2. Then use manual function calling where you handle the function call and as part of your handling you prompt the user. Take a look at the samples here FunctionCalling.cs

[deepinderdeol]

@markwallace-microsoft Thanks for the suggestion.
I have looked at the transform example before and it seems useful, but for our scenario the user request could contain all the information - it would seem odd to ask the user again for the information they already specified.

Just curious, is it a known issue with function calling that even though a parameter is marked required, the LLMs can ignore that and invent a value for an argument in some scenarios?

Additional information - we have seen this issue mostly for functions with multiple parameters. If there is only one parameter then behavior is pretty consistent and the user is prompted for missing information.

Example:

• ApplySymbology(string layerName, string fieldName)
• User request: apply symbol to states layer
• The LLM correctly sets layerName = 'states'
• However, it also specifies a fieldName on its own. Sometimes the fieldName it comes up with works because it is a reasonable assumption like "Name" or "StateName", but this is not good because the actual field name could be say "State_Name" or, more importantly, the field that the user wanted to use could be something that is not very obvious.