Type: url
Original: https://arxiv.org/pdf/2606.17799
Status: generated
Summary
A position paper from Tessl arguing that current coding benchmarks measure the wrong object: a coding agent in practice is not a model but a system harness — a composite of models, harnesses, contexts, environments and feedback signals, any one of which can move a benchmark score by margins comparable to those between adjacent model generations. It sets out three symptoms of the misalignment (scores conflate the model with the rest of the harness, grading against a single reference solution penalises equally valid alternatives, and the absence of component-level signal makes the end-to-end score hard to iterate on) and proposes a structural remedy for each. It draws throughout on NS2, an issue-driven system harness the authors built and open-sourced.