Has anyone had any success with evaluating the impact of using Generative AI tools such as GitHub's Copilot on the productivity or performance impact on developers? I see a lot of qualitative discussions about how developers say they are more productive, but how are you measuring that impact?

13.3k views53 Upvotes9 Comments

IT Manager, Self-employed
One of the things we're looking at is the number of commits, PRs, deploys that are being done in a repo before and after devs start using Copilot. We're also considering test coverage % and post deployment issues. 
Chief of DevOps and Partner in Healthcare and Biotech, 1,001 - 5,000 employees
Here is a link to a study where they recruited 95 professional developers, split them randomly into two groups, and timed how long it took them to write an HTTP server in JavaScript.
Research: quantifying GitHub Copilot’s impact on developer productivity and happiness - The GitHub Blog


The group that used GitHub Copilot had a higher rate of completing the task (78%, compared to 70% in the group without Copilot).
The striking difference was that developers who used GitHub Copilot completed the task significantly faster–55% faster than the developers who didn’t use GitHub Copilot. Specifically, the developers using GitHub Copilot took on average 1 hour and 11 minutes to complete the task, while the developers who didn’t use GitHub Copilot took on average 2 hours and 41 minutes. These results are statistically significant (P=.0017) and the 95% confidence interval for the percentage speed gain is [21%, 89%].
3 4 Replies
Global Chief Cybersecurity Strategist & CISO in Healthcare and Biotech, Self-employed

Thanks for sharing. Interesting study 🤔

Chief of DevOps and Partner in Healthcare and Biotech, 1,001 - 5,000 employees

🤗 you are welcome. 

IT Manager, Self-employed

Thanks Romano. Yes, I had seen that study (really the only one I found that had actual metrics). It's a start, but really that's a fairly artificial example, since in real life we would never set a bunch of our developers up to all code the same thing. I was hoping that someone had done a live before and after measurement of developer productivity. The search continues . . .

CTO in Consumer Goods, 11 - 50 employees
We have a team of 30 developers using Copilot over the last five months. Exclusively Java and TypeScript developers building enterprise software. We’ve seen around 5% productivity gains, which is pretty much what we expected. It’s very good for well documented APIs and boilerplate code. It’s pretty much a wash with proprietary business logic.
3 2 Replies
IT Manager, Self-employed

Thanks Matthew. How was the 5% gain calculated? We're really looking to see if there is a way to actual measure the impact short of doing a survey and asking the devs if they thought they were more productive.

CTO in Consumer Goods, 11 - 50 employees

We measure the cycle time (from feature start until merge) across all of our development teams. 


Content you might like

Modbus (widely used protocol in industrial automation and control systems)13%

OPC UA (protocol for machine-to-machine communication that is designed for use in industrial automation and control systems)48%

MQTT (lightweight messaging protocol that is designed for use in low-bandwidth, high-latency networks)21%

DDS (real-time publish-subscribe communication protocol that is designed for use in distributed systems)10%

AMQP (messaging protocol that is designed for use in distributed systems)2%

LoRaWAN (long-range radio-wide area network used for IoT, smart cities, and industrial applications)1%

Proprietary protocols (please, comment)4%


674 views1 Upvote

We are not doing regression testing10%

25% manual, 75% automated50%

50% manual, 50% automated27%

100% manual, 0% automated8%

Don't know2%


1.6k views3 Upvotes2 Comments

CTO in Software, 201 - 500 employees
Without a doubt - Technical Debt! It's a ball and chain that creates an ever increasing drag on any organization, stifles innovation, and prevents transformation.
Read More Comments
40.8k views131 Upvotes319 Comments