Spec-driven test generation is a way of prompting an LLM-based coding agent to write tests: rather than asking for tests directly, the agent is instructed first to reason about — and explicitly document — the code’s pre-conditions, post-conditions and undefined behaviors. The resulting intermediate, semi-formal specification is then used as what its proposers call a cognitive scaffold to guide the test generation that follows. The term is proposed in Grounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test Generation.
Usage
What the technique adds is an artifact: a description of the code’s contract, produced and written down before any test is. Its motivation is a failure attributed to direct prompting — that an agent asked straight for tests can fail to reason about the code and its underlying contracts, and so miss edge cases and behavioral boundaries that bear on test quality.
When It Applies
It is proposed for agentic test generation against existing code, and evaluated in that setting on production bugs from Google, where the reported gains over a conventional test-generation agent are 9.8 percentage points of bug detection rate and 2.5 percentage points of branch coverage. A model-judged comparison is reported to prefer its suites in 77.8% of cases against that baseline, and in 56.7% of cases against human-authored tests.
What the term names here is specific: the specification is written by the agent, about code that already exists, and serves to ground the tests generated from it rather than to drive an implementation. See Spec-driven development for the broader practice that the proposing paper’s venue places this in.
Related Terms
Spec-driven development, LLM-as-a-Judge, Test Design, Characterization Test