Type: url
Original: https://arxiv.org/pdf/2507.21504
Status: generated
Language: en
Summary
A KDD ‘25 survey from SAP Labs on how LLM-based agents are evaluated. It proposes a two-dimensional taxonomy that organizes existing work by evaluation objectives (agent behavior, capabilities, reliability, and safety and alignment) and by evaluation process (interaction mode, evaluation data, metrics computation methods, tooling and evaluation contexts), and catalogs representative metrics and benchmarks for each. It also highlights enterprise-specific challenges often overlooked in research — role-based access control, reliability guarantees, long-horizon interactions and policy compliance — and calls for holistic, more realistic, automated and time- and cost-bounded evaluation.