CoverUp is a tool for generating high-coverage regression tests for Python code with a large language model. It measures the coverage of an existing test suite, identifies the functions, methods or classes containing lines or branches that do not execute, and prompts the LLM for pytest tests aimed at exactly that uncovered code. Tests that pass and increase coverage are kept; for the rest, CoverUp continues the chat with the LLM, pointing out errors or the coverage still missing and asking for improvements.
It was introduced in CoverUp: Effective High Coverage Test Generation for Python by researchers at the University of Massachusetts Amherst, and is published on GitHub and archived on Zenodo. Its authors state that it can be applied to any Python package, and that its approach is independent of the LLM, although it was developed and evaluated with OpenAI’s GPT-4, GPT-4 Turbo and GPT-4o models.
Capabilities
CoverUp uses the SlipCover coverage analyzer and Python’s abstract syntax tree to split code lacking coverage into segments, by default up to 50 lines, keeping a few lines of an enclosing class definition and the original comments for context. Each initial prompt assigns the LLM the persona of an expert Python test-driven developer, names the file, states which lines and branches do not execute, tags those lines with their numbers, adds import statements generated by static analysis, and asks for complete test scripts with assertions and no state pollution. A get_info tool function lets the LLM request the definition of any name in the excerpt, returning abbreviated class or module definitions with omitted bodies marked by Python’s ....
Generated tests are run several times with pytest-repeat to surface flaky tests. To handle tests that pollute shared state, the tool by default runs each test in isolation using the Unix fork() system call, and alternatively can search for and disable polluting tests or disable failing tests, leaving disabled tests available for review. It can install Python modules that generated tests need and record them in requirements.txt format. Prompting and verification run asynchronously, with requests spread out by a leaky-bucket rate limiter to stay within OpenAI’s limits, and progress is checkpointed so a run can be resumed. The whole process can be repeated to refocus the LLM on segments missed in earlier passes.
Adoption & Ecosystem
The introducing paper evaluates CoverUp against CodaMosa and MuTAP on benchmark suites derived from open-source Python projects and from HumanEval, reporting higher coverage than both. Its authors describe it as developed concurrently with other LLM-based test generators such as Meta’s TestGen-LLM, which also rejects tests that fail or do not add coverage but discards them rather than continuing the conversation.