Type: url
Original: https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study-paper.pdf
Status: generated
Summary
A randomized controlled trial by METR measuring what February–June 2025 frontier AI tools did to the productivity of experienced open-source developers. Sixteen developers completed 246 real issues on repositories they averaged five years of work on, with each issue randomly allowing or disallowing AI; allowing AI increased completion time by 19%, against a 24% speedup the developers forecast beforehand, a 20% speedup they still estimated afterwards, and 39% and 38% speedups forecast by economics and machine-learning experts. Labelled screen recordings show the time moving from coding and searching into prompting, waiting and reviewing, and the paper examines twenty candidate explanations, finding evidence for five — over-optimism, developer familiarity with the repositories, repository size and complexity, low AI reliability, and implicit repository context — while cautioning that the result is specific to this setting.