The paper studies the engineering of AI-assisted coding tools such as Claude Code, Codex and Gemini CLI. It argues that while these tools promise significant productivity gains, building them — at the intersection of traditional software engineering, AI system design and human-computer interaction — is fraught with unique and poorly understood challenges, and presents itself as the first empirical study of the engineering pitfalls involved.
The authors systematically and manually analyse over 3.8K publicly reported bugs in the open-source GitHub repositories of those three tools. Using an open-coding methodology, they examine each issue description together with the associated user discussion and developer responses, and categorize every bug by type, location, root cause and observed symptoms. This annotation is used to characterize common failure patterns and recurring engineering challenges, which the authors offer as a roadmap for developers designing more reliable AI coding assistants.
Key Points
- More than 67% of the bugs studied are related to functionality.
- In terms of root causes, 36.9% of the bugs stem from API, integration or configuration errors.
- The most commonly observed symptoms are API errors (18.3%), terminal problems (14%) and command failures (12.7%).
- The bugs predominantly affect the tool invocation (37.2%) and command execution (24.7%) stages of the system workflow.
- All of these figures are drawn from publicly reported issues in the repositories of three tools — Claude Code, Codex and Gemini CLI.
Notes
The paper was submitted to arXiv on 21 March 2026 and is filed under Software Engineering (cs.SE).