IFEval-FC is a benchmark for precise instruction following in Function calling. It was introduced in Instruction-Following Evaluation in Function Calling for Large Language Models and adapts the “verifiable instructions” of the IFEval benchmark to the function-calling setting: each instruction is embedded in the description field of a parameter in a JSON schema, and whether the model’s call follows it is checked algorithmically.
Contents
The benchmark offers 750 test cases, each consisting of a function with an embedded format instruction for one of its input parameters and a corresponding user query. Its instructions are drawn from 19 distinct types of verifiable instruction, which the author organizes into seven major categories according to the nature of the constraint — for example requiring a Python list format, restricting text to Cyrillic or Greek script, or controlling word or comma counts. Some instruction types were adapted from IFEval and others, such as Cyrillic/Greek and Python list format, were introduced for this benchmark. Each response is scored 1 if the instruction is followed and 0 otherwise.
Provenance
Part of the functions were taken from the Berkeley Function Calling Leaderboard (BFCL) and enhanced with format constraints; the rest were generated with GPT-5 across 80 curated domains, each required to include one free-form natural-language string parameter with no enum or format field so that any format constraint could be applied to it. Five conversational user queries were generated per function from the original, constraint-free functions, so that it is the model’s job during the call to transform a value into the required format. Tasks that an ensemble of LLMs failed on all five queries, or answered correctly on all five with ease, were omitted, and trivially easy instruction types such as all-uppercase or all-lowercase were excluded from the final evaluation. The code and data are publicly available on GitHub.
Use
The introducing paper evaluates GigaChat, Anthropic and OpenAI models on the benchmark and reports that no evaluated model surpassed 80% accuracy.