NL2Repo-Bench is a benchmark, introduced in NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents, that is designed to evaluate the long-horizon repository generation ability of coding agents — whether they can sustain coherent reasoning, planning and execution long enough to build a complete software repository rather than a localized piece of code.
Contents
Each task gives an agent only a single natural-language requirements document and an empty workspace. From that, the agent must autonomously design the architecture, manage dependencies, implement multi-module logic, and produce a fully installable Python library. Results are reported as test pass rates, which the paper describes as making the benchmark a verifiable testbed.
Use
NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents evaluates state-of-the-art open- and closed-source models on the benchmark and reports that even the strongest agents achieve below 40% average test pass rates and rarely complete an entire repository correctly. It identifies failure modes including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps.