Knowledge sharing • 1 min read •
The Impact of AI on Software Development Speed
Does AI really accelerate software development? A recent study by researchers at METR challenges this widely held belief. Their findings reveal that, under certain conditions, experienced developers actually completed tasks more slowly when using LLMs than when working without AI assistance.
Both industry experts and software developers generally agree that Large Language Models (LLMs), such as ChatGPT, can accelerate software development. However, a recent study conducted by researchers at METR paints a more nuanced picture.
The researchers recruited 16 experienced software engineers and assigned them 246 real-world development tasks across projects they had worked on for an average of five years, meaning they were already familiar with the codebases. Before the experiment, participants who had not previously used LLMs received training on Cursor, an AI-powered development environment. For each task, it was randomly determined whether the developer could use an LLM or had to complete the task without AI assistance.
Both before and after completing the tasks, participants estimated how much faster or slower they expected to be when using AI. The same question was also posed to external experts. Nearly everyone predicted that LLMs would improve productivity—but the actual results told a different story.
The study found that developers completed their tasks 19% more slowly on average when they were allowed to use LLMs. Several factors may explain this outcome. Large language models often struggle to fully understand extensive codebases, the generated code is not always reliable and requires verification, and generating responses itself introduces additional overhead.
These findings do not suggest that LLMs cannot improve software development productivity. Instead, they highlight that their benefits depend heavily on context. In environments involving large codebases and highly experienced developers, the productivity gains may be smaller than expected—or may even disappear entirely.
