Type: url
Original: https://arxiv.org/pdf/2608.26623
Status: generated
Language: en
Summary
This ServiceNow AI paper introduces AgentJudgeBench, a benchmark of 3,808 synthetic, BFCL-style tool-calling records spanning six DAG topologies and three difficulty tiers, built to measure how reliably LLM judges score agentic tool-calling against a deterministic programmatic reference, with and without the ground-truth trace. Across five generators and six judges it finds that judge alignment falls monotonically with difficulty (faster without ground truth), that all judges converge to a 77–82% band on hard queries without ground truth, that showing ground truth lowers alignment for GPT-5.4 and Gemini-2.5-Pro, and that structured per-metric rubrics help while chain-of-thought and temperature have negligible effect.