Type: url
Original: https://arxiv.org/abs/2310.06770
Status: generated
Summary
The arXiv abstract page for SWE-bench, an evaluation framework of 2,294 software engineering problems drawn from real GitHub issues and their corresponding pull requests across 12 popular Python repositories. A model is given a codebase and a description of an issue and must edit the codebase to resolve it, which the authors argue demands coordinating changes across multiple functions, classes and files, interacting with execution environments and processing extremely long contexts. The reported results are low: state-of-the-art proprietary models and the authors’ fine-tuned SWE-Llama resolve only the simplest issues, with the best-performing model, Claude 2, solving 1.96%.