Type: url
Original: https://arxiv.org/pdf/2511.05459
Status: generated
Summary
Introduces SWE-Compass, a benchmark of 2,000 verified instances curated from real GitHub pull requests spanning eight task types, eight programming scenarios and ten programming languages, each paired with a reproducible containerised execution environment. Ten state-of-the-art LLMs are evaluated under two agentic frameworks, SWE-Agent and Claude Code, with Claude-Sonnet-4 ranking first under both (32.9% macro-average under Claude Code and 31.8% under SWE-Agent) and most scores clustering in the low-to-mid twenties. A post-hoc LLM-as-judge failure analysis over 600 failed trajectories per model attributes more than 60% of failures to requirement misinterpretation and to incomplete solutions with side effects, while technical knowledge gaps account for only 5-8%.