Type: url
Original: https://arxiv.org/pdf/2509.18420
Status: generated
Language: en
Summary
A short preprint introducing IFEval-FC, a benchmark inspired by IFEval that tests whether LLMs follow verifiable format instructions embedded in the parameter descriptions of JSON function schemas (for example “must be lowercase” or “should not include punctuation”), a dimension existing function-calling benchmarks such as BFCL, τ2-Bench and ACEBench do not test. It offers 750 test cases, each a function with a format constraint on one parameter plus a user query, scored fully algorithmically. The results show that even state-of-the-art proprietary models such as GPT-5 and Claude Opus 4.1 frequently fail such basic formatting rules, and no evaluated model surpassed 80% accuracy.