Type: url
Original: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
Status: partial
Summary
An Anthropic engineering post (Jan 2026) on designing automated evaluations for AI agents. It defines the components of an agent eval (tasks, trials, graders, transcripts, outcomes, evaluation and agent harnesses, suites), compares code-based, model-based and human graders, distinguishes capability from regression evals, covers coding, conversational, research and computer-use agents, introduces pass@k and pass^k for non-determinism, and gives an eight-step roadmap from zero to trustworthy evals alongside how evals fit with production monitoring, A/B tests and human review.