This paper is a survey of evaluation methods for LLM-based agents, published in Findings of the Association for Computational Linguistics: ACL 2026. Its starting position is that LLM-based agents represent a paradigm shift in AI: systems that plan, reason and use tools autonomously while interacting with dynamic environments. The authors describe their survey as the first comprehensive treatment of how such agents are evaluated.
The survey organizes the field of agent evaluation into five perspectives. The first covers the core LLM capabilities that agentic workflows depend on, naming planning and tool use as examples. The second covers application-specific benchmarks, including those for web agents and for SWE (software engineering) agents. The third covers the evaluation of generalist agents, the fourth analyses the core dimensions along which agent benchmarks are constructed, and the fifth surveys the evaluation frameworks and tools available to agent developers.
From that analysis the authors report a trend in the field toward more realistic and more challenging evaluations, supported by benchmarks that are continuously updated rather than fixed at release. They also name the gaps they consider most critical for future work.
Key Points
- The survey presents itself as the first comprehensive survey of evaluation methods for LLM-based agents.
- It analyses agent evaluation across five perspectives: core LLM capabilities needed for agentic workflows, application-specific benchmarks, evaluation of generalist agents, analysis of agent benchmarks’ core dimensions, and evaluation frameworks and tools for agent developers.
- Web agents and SWE agents are named as the application-specific benchmark categories the survey examines.
- The authors report a trend toward more realistic, more challenging evaluations built on continuously updated benchmarks.
- They identify cost-efficiency, safety and robustness as the dimensions current agent evaluation assesses least well.
- They also call for fine-grained, scalable evaluation methods, which they treat as an open problem rather than a solved one.
Notes
The gaps the authors name — cost-efficiency, safety, robustness, and fine-grained scalable evaluation — are stated as directions future research must address rather than as findings the survey itself settles. The paper appears in the Findings track of ACL 2026 and is published by the Association for Computational Linguistics under DOI 10.18653/v1/2026.findings-acl.1330.