The post (“AutoDev CLI: building a quality-assurance and validation architecture for AI agents generated by AI agents”) was published on 12 November 2025 by Phodal Huang, the author of AutoDev (unit-mesh/auto-dev). It explains that the project’s CLI, built on the new AutoDev MPP (Multiplatform Paradigm) core, was not designed to follow the fashion for coding CLIs but to fix a testing problem in the previous generation of AutoDev, which was built on the IntelliJ UI framework. It then walks through how AI coding tools were used, prompt by prompt, first to build the CLI and then to generate a test framework for the AutoDev coding agent itself.
The author’s summary is that this marks a shift from “AI generating code” to “AI verifying code” and on to “AI generating the systems that test AI”, and that in an era of uncertain AI output the CLI gives back engineering determinism.
Key Points
- The author lists why the IntelliJ-based agent was hard to test: agent logic was tightly bound to UI events, context state such as project, editor and selection depended on IDE kernel objects, context dependencies such as AST and dependency relations were not separated out, and much logic ran on the IDE runtime and could not easily run in CI/CD.
- The CLI was started by prompting AI coding tools to build a terminal UI on Node.js with the Ink library, integrated with the real
mpp-coreagent and organised by Kotlin Multiplatform practice; the author says the rest was left to Augment. - Because the new architecture shares the same CodingAgent code and renderer across targets, the author reasons that problems seen in the CLI will very probably also appear on desktop and mobile, so the CLI becomes a fast place to reproduce and fix them.
- The CLI’s log output is treated as the input for the next fix: with enough logging, an AI agent can build, run, inspect the output, change the code and repeat until it reaches the goal or gives up.
- After collecting scenarios, the author had AI extract a dedicated test framework focused on three things — the prompts generated, the tools called, and the code changes produced — with analyzers for each, a test engine, a console reporter and a scenario builder.
- Test cases are then generated by AI with expectations about behaviour, such as a minimum number of file reads and writes or specific files being created or modified.
- The framework was wired into a GitHub Actions workflow with a DeepSeek key from repository secrets, with the author asking for a pass threshold of at least 80% and good reporting.
- The three capabilities the author claims for the resulting architecture are verifiable generation with traceable context, logs and behaviour; a test framework that AI generates and maintains itself; and portable environments through the unified CLI and MPP structure.
Context
The post is a first-hand account by the tool’s own author and reports no measured results. It extends the author’s earlier writing on using Vibe Coding to build AI agents, and belongs with this wiki’s material on treating agent behaviour as something to test, such as Trajectory Evaluation.