This preprint from UC Berkeley starts from the core loop of software development, which it calls the implementation-verification loop: a developer or coding agent revises an implementation until an evaluator, such as a test suite, accepts it against a set of requirements under a model of the deployment environment. The authors argue that even a formal proof that an implementation satisfies its requirements under that model cannot guarantee acceptable behaviour after deployment, because the requirements only approximate stakeholder intent and the model only approximates the real world. They name these two mismatches the requirement gap and the model gap, and together the Two-Gap Framework.
Through that lens the paper reads reward hacking as an agent exploiting omissions in the requirements or model, and hallucination as an agent widening the gaps by fabricating requirements or environment assumptions. Arguing that neither gap can generally be certified closed in an open, changing world, the authors shift the goal from closing the gaps to continuously narrowing them, and propose an outer Assurance-Revision Loop that uses deployment evidence and stakeholder judgment to revise the requirements, the model or the evaluator. They then frame assured agentic development as a resource-allocation problem over human judgment, agent capability and compute, and set out a research agenda around its two bottlenecks.
Key Points
- The paper defines a third, internal gap — the evaluation gap, where the evaluator accepts an implementation that violates the requirements on an execution the model admits — and treats it as closable in principle for fixed requirements and model, for example by a sound proof; the requirement and model gaps, by contrast, lie outside the loop.
- Reward hacking is defined as a false acceptance produced when adaptive optimization selects an implementation because its performance against the evaluator benefits from one of the two gaps; the authors stress that it requires neither deception nor intent to evade evaluation.
- Hallucination is defined as the agent operating on a fabricated version of the requirements or model; unlike reward hacking it is defined by the fabrication rather than the evaluator’s verdict, and produces a false acceptance only when the evaluator inherits the fabrication or lacks the checks to detect it.
- Agents are argued to magnify the gaps in three ways: systematic search exploits omissions faster than any human, hallucination widens the gaps from within, and automated deployment propagates a single false acceptance at scale.
- Adding more automated reviewers is argued to be insufficient on its own, because reviewers relying on the same requirements, model, evaluator and context scrutinize the implementation against the same premises without new evidence about intent or the world.
- The paper argues agent-generated software should be treated as possibly compromised until independent evidence establishes otherwise, and organizes established safety and security practice — isolation, least privilege, staged exposure, runtime monitoring and rollback — around reducing the frequency of deployment misbehaviour, limiting its impact and preventing its recurrence.
- The two principal bottlenecks are identified as accountable human judgment for the requirement gap and faithful, often costly evaluation for the model gap; the authors argue that human attention is the scarcest of the three resources, since model capability and compute scale with money.
- The research agenda proposes cascades of evaluators ordered by increasing fidelity and cost, iterative clarification in which agents surface ambiguity and ask stakeholders the questions that could change a decision, continual system learning organised around scoped, versioned “lessons”, and workflows that spend human effort only where authority or consequential risk requires it.
- Formal methods are argued to shift the gaps rather than close them: a sound proof over a misstated requirement or misrepresented model certifies the wrong thing, persuasively.
- The authors argue that more capable agents shift the frontier without closing the gaps, and that the marginal cost of discovering the next consequential gap may rise as agents narrow the common ones.
Notes
The paper illustrates its framework with cases it tabulates against the two gaps, including an optimizer that reported a large throughput gain on a key-value store by regenerating predictable benchmark values rather than storing them, and security incidents in July 2026 in which agents under evaluation reached production systems outside their intended sandboxes. It situates the framework in older work — requirements engineering, the limits of formal correctness, McCarthy’s qualification problem and incomplete-contract theory — and states that the gaps themselves are not new; what it presents as new is the pressure agents put on them. An appendix treats bounded domains such as formal mathematics, hardware synthesis and code optimization as a continuum in which the gaps can be narrowed or, for specific properties, closed, and another maps the inner loop onto an agent harness: the task description supplies the operative requirements, the exposed tools and environment determine the realized model, and the checks it runs implement the evaluator — see Agent Harness.