This paper proposes and evaluates Spec-Driven Test Generation, an intervention in how a coding agent is prompted for test generation. Its starting observation is that LLM-based agents, though they have outperformed many classical approaches and scaled to repository-level tasks such as test generation, can fail to reason about the code and its underlying contracts when asked directly for tests — missing edge cases and behavioral boundaries that affect test quality. The proposed remedy is an intermediate step: instruct the agent first to reason about, and explicitly write down, the code’s pre-conditions, post-conditions and undefined behaviors, and let that semi-formal specification act as what the authors call a cognitive scaffold for the test generation that follows.
The evaluation is on production bugs from Google, comparing the spec-driven agent against a traditional test-generation agent baseline, and is reported on two automatic measures plus a model-judged qualitative comparison. The paper is to appear in the Proceedings of the 1st International Workshop on Specification-Driven Development Life Cycle (SpecOps 2026), co-located with SPLASH 2026. It was submitted on 17 August 2026 and last revised on 21 August 2026 (v2, the version summarized here). It carries the arXiv DOI 10.48550/arXiv.2608.17177 and a related DOI at 10.1145/3842652.3843195, and is released under CC BY 4.0.
Key Points
- The stated failure mode being addressed is that an agent prompted directly for tests can fail to reason about the code and its underlying contracts, missing edge cases and behavioral boundaries.
- The proposed method has the agent produce an explicit intermediate artifact — documented pre-conditions, post-conditions and undefined behaviors — before generating tests.
- On production bugs from Google, the spec-driven agent is reported to deliver a 9.8 percentage point improvement in bug detection rate, with a reported p-value of 0.0352.
- On the same evaluation it is reported to deliver a 2.5 percentage point improvement in branch coverage, with a reported p-value of 0.0034.
- Using LLM-as-a-Judge, the paper reports its test suites judged superior to the baseline’s in 77.8% of cases and superior to human-authored tests in 56.7% of cases.
- The judged improvements are reported along three named dimensions: following best practices, readability, and edge-case coverage.
- Both the effect sizes and the judged comparisons rest on one evaluation set — production bugs from a single organization — and the qualitative comparison is model-judged rather than human-judged.
Notes
The paper’s venue places it in the Spec-driven development line of work: it is to appear at a workshop dedicated to the specification-driven development life cycle. What the abstract itself sets out is specific — the specification is produced by the agent, about code that already exists, and its purpose is to ground the tests the agent then writes.