At a glance
- Source
- Becker, Rush, Barnes and Rein (METR) (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR, published 10 July 2025 (arXiv 2507.09089); follow-up published 24 February 2026. metr.org · Not peer-reviewed: other researchers have not yet checked it for a journal.
- How it was done
- Each task was assigned by chance to be done with AI tools allowed or not allowed. The researchers recorded the developers’ screens and timed the work, and asked the developers for their own estimates before and after. Randomised experiment. People were assigned by chance to work with or without the tool, and the two groups were compared.
- Who took part
- 16 experienced open-source developers working on 246 real tasks. The 2026 follow-up had 57 developers and more than 800 tasks.
- Where
- Mature open-source software projects the developers had worked on for years, with AI tools available in early 2025.
“when developers use AI tools, they take 19% longer than without”
What it found
- Measured time. Tasks with AI allowed took 19% longer than tasks without it. The likely range runs from 2% to 39% longer.
- What the developers expected. Before starting, they forecast that AI would cut their time by 24%.
- What the developers believed afterwards. Once the work was done, they still thought AI had cut their time by about 20%.
The developers’ own estimates pointed the wrong way, not only by the wrong amount. The screen recordings, not the developers’ impressions, show the gap.
The 2026 follow-up
METR, a non-profit that tests AI systems, ran the study again from August 2025 with newer tools: 57 developers and more than 800 tasks. Ten of them had taken part in 2025.
- For the returning developers, tasks with AI took an estimated 18% less time. The likely range runs from 38% less to 9% more.
- For the new developers, tasks with AI took an estimated 4% less time. The likely range runs from 15% less to 9% more.
Both ranges include no change at all. METR calls the 2026 figures unreliable. Some developers declined to take part because they did not want to work without AI. Between 30% and 50% said they held back tasks they did not want to do without AI. Time records were unreliable for developers who ran several AI agents at once. METR thinks developers were probably faster with AI in early 2026 than in early 2025, but says its data are “only very weak evidence for the size of this increase”.
Some coverage reports the 2026 result as an established 18% gain. By METR’s own account, neither 2026 estimate can be told apart from no change.
When the result holds
The developers were experts working on code they knew well. That is the setting where AI might help least. The study says nothing about novices or unfamiliar work.
The gap between belief and measurement is the part most likely to carry over to other jobs. The 2026 follow-up adds a lesson on method: once people are used to a tool, many will not work without it, so a comparison group without the tool has to be set up before rollout.
Limits
- A small sample, one occupation and tools from early 2025. METR now marks the 2025 results as out of date.
- Not published in a peer-reviewed journal.
Practices that use this study
This study supports: