Type: url
Original: https://arxiv.org/pdf/2508.07575
Status: generated
Summary
This paper from Ant Group introduces MCPToolBench++, a benchmark of about 1.5K queries for evaluating how LLMs and AI agents call Model Context Protocol tools, built automatically from tool schemas collected from MCP marketplaces and covering six domains with single- and multi-step calls. It evaluates five models with AST, AST DAG accuracy, Pass@K and tool call success rate metrics, finding that no model leads every category and that tool-selection scores and execution success do not always agree. It also analyses the root causes of failed MCP tool calls.