Type: url
Original: https://www.anthropic.com/engineering/infrastructure-noise
Status: partial
Summary
Anthropic reports that infrastructure configuration alone can move scores on agentic coding benchmarks by several percentage points: running Terminal-Bench 2.0 under six resource configurations, the gap between the most- and least-resourced setups was 6 percentage points, and a smaller effect was also seen on SWE-bench. The post explains how container resource enforcement and other factors such as time limits act as confounders, and recommends that evals specify both a guaranteed allocation and a separate hard limit per task, and that leaderboard differences below 3 percentage points be treated with skepticism until configurations are documented and matched.