This arXiv preprint introduces SWE-Agent and makes an argument about why it works. Its starting observation is that language model agents are increasingly used to automate complicated tasks in digital environments, and its proposal is to treat those agents as a new category of end users — ones with their own needs and abilities, who would benefit from specially-built interfaces to the software they use, just as humans benefit from powerful applications such as integrated development environments when doing complex work like software engineering.
From that premise the paper investigates how interface design affects the performance of language model agents, and introduces SWE-agent as the result of that investigation: a system that facilitates LM agents in autonomously using computers to solve software engineering tasks. The component the paper singles out is SWE-agent’s custom Agent-Computer Interface (ACI), which the authors state significantly enhances an agent’s ability to create and edit code files, navigate entire repositories, and execute tests and other programs.
The authors evaluate SWE-agent on SWE-Bench and on HumanEvalFix, reporting state-of-the-art performance on both with pass@1 rates of 12.5% and 87.7% respectively — figures they describe as far exceeding the previous state of the art achieved with non-interactive language models. The paper closes by offering insight into how the design of the ACI can affect agents’ behavior and performance.
The preprint is filed under Software Engineering (cs.SE), with cross-listings to Artificial Intelligence (cs.AI), Computation and Language (cs.CL), Human-Computer Interaction (cs.HC) and Machine Learning (cs.LG). It was first submitted on 6 May 2024 and last revised on 11 November 2024 as version 3, and carries the arXiv-issued DOI 10.48550/arXiv.2405.15793. Code, data and a demo are stated to be available at https://swe-agent.com.
Key Points
- The paper’s framing claim is that LM agents represent a new category of end users with their own needs and abilities, and therefore would benefit from interfaces built specifically for them.
- It introduces SWE-agent, a system that facilitates LM agents in autonomously using computers to solve software engineering tasks.
- Its named mechanism is a custom agent-computer interface (ACI), which the authors state significantly enhances an agent’s ability to create and edit code files, navigate entire repositories, and execute tests and other programs.
- The reported results are a pass@1 rate of 12.5% on SWE-bench and 87.7% on HumanEvalFix, described as state of the art on both and as far exceeding what had been achieved with non-interactive LMs — figures measured on these two benchmarks in this version of the paper, not standing claims about either benchmark.
- The paper also offers, as a contribution of its own, insight into how the ACI’s design affects agent behavior and performance.
Notes
The title states the paper’s thesis — that agent-computer interfaces enable automated software engineering — so the contribution is a design claim about interfaces as much as a system. Its evaluation rests on two benchmarks, SWE-bench and HumanEvalFix. The preprint went through three versions between May and November 2024.