Type: url
Original: https://arxiv.org/pdf/2410.06992
Status: generated
Summary
An empirical study from York University of the SWE-bench benchmark, which manually compared the 251 SWE-Agent+GPT-4 patches that passed all tests against the gold patches and found that 32.67% involved solution leakage (the fix was given in the issue report or its comments) and 31.08% passed only because of weak tests, lowering that system’s resolution rate from 12.47% to 3.97%; the same problems were found in SWE-bench Lite and SWE-bench Verified. To address this the authors built SWE-Bench+, 548 task instances from issues created after the models’ training cut-off dates and without solutions in their reports, on which the resolution rates of the evaluated systems drop sharply while weak tests remain a problem.