Has anyone had any success with evaluating the impact of using Generative AI tools such as GitHub's Copilot on the productivity or performance impact on developers? I see a lot of qualitative discussions about how developers say they are more productive, but how are you measuring that impact?

Disruptive & Emerging TechnologiesSoftware Development
13.3k views53 Upvotes9 Comments
IT Manager, Self-employed
One of the things we're looking at is the number of commits, PRs, deploys that are being done in a repo before and after devs start using Copilot. We're also considering test coverage % and post deployment issues. 
Chief of DevOps and Partner in Healthcare and Biotech, 1,001 - 5,000 employees
Here is a link to a study where they recruited 95 professional developers, split them randomly into two groups, and timed how long it took them to write an HTTP server in JavaScript.
Research: quantifying GitHub Copilot’s impact on developer productivity and happiness - The GitHub Blog

Result: 

The group that used GitHub Copilot had a higher rate of completing the task (78%, compared to 70% in the group without Copilot).
The striking difference was that developers who used GitHub Copilot completed the task significantly faster–55% faster than the developers who didn’t use GitHub Copilot. Specifically, the developers using GitHub Copilot took on average 1 hour and 11 minutes to complete the task, while the developers who didn’t use GitHub Copilot took on average 2 hours and 41 minutes. These results are statistically significant (P=.0017) and the 95% confidence interval for the percentage speed gain is [21%, 89%].
3 4 Replies
Global Chief Cybersecurity Strategist & CISO in Healthcare and Biotech, Self-employed

Thanks for sharing. Interesting study 🤔

1
Chief of DevOps and Partner in Healthcare and Biotech, 1,001 - 5,000 employees

🤗 you are welcome. 

IT Manager, Self-employed

Thanks Romano. Yes, I had seen that study (really the only one I found that had actual metrics). It's a start, but really that's a fairly artificial example, since in real life we would never set a bunch of our developers up to all code the same thing. I was hoping that someone had done a live before and after measurement of developer productivity. The search continues . . .

1
CTO in Consumer Goods, 11 - 50 employees
We have a team of 30 developers using Copilot over the last five months. Exclusively Java and TypeScript developers building enterprise software. We’ve seen around 5% productivity gains, which is pretty much what we expected. It’s very good for well documented APIs and boilerplate code. It’s pretty much a wash with proprietary business logic.
3 2 Replies
IT Manager, Self-employed

Thanks Matthew. How was the 5% gain calculated? We're really looking to see if there is a way to actual measure the impact short of doing a survey and asking the devs if they thought they were more productive.

1
CTO in Consumer Goods, 11 - 50 employees

We measure the cycle time (from feature start until merge) across all of our development teams. 

1

Content you might like

Is the Double Diamond Design Model Still Relevant? In today’s world where code can be pushed multiple times a day, and prototypes quickly tested, the rigid phases of the Double Diamond can feel constraining. The model’s omission of execution for three-quarters of the process could create significant risk. Do you think is it time to reconsider this design approach?

End-User Services & CollaborationDisruptive & Emerging TechnologiesStrategy & Architecture
245 views

According to your experience, which protocol is most commonly used for communication in the Industrial Internet of Things (IIoT)?

Disruptive & Emerging TechnologiesInfrastructure

Modbus (widely used protocol in industrial automation and control systems)13%

OPC UA (protocol for machine-to-machine communication that is designed for use in industrial automation and control systems)48%

MQTT (lightweight messaging protocol that is designed for use in low-bandwidth, high-latency networks)21%

DDS (real-time publish-subscribe communication protocol that is designed for use in distributed systems)10%

AMQP (messaging protocol that is designed for use in distributed systems)2%

LoRaWAN (long-range radio-wide area network used for IoT, smart cities, and industrial applications)1%

Proprietary protocols (please, comment)4%


135 PARTICIPANTS
674 views1 Upvote

At your organization, how much regression testing for new app features is done manually vs. how much is automated?

EngineeringTechnical Product ManagementSoftware Development+8 more

We are not doing regression testing10%

25% manual, 75% automated50%

50% manual, 50% automated27%

100% manual, 0% automated8%

Don't know2%


698 PARTICIPANTS
1.6k views3 Upvotes2 Comments

If you had a magic wand - what's the #1 daily business challenge you'd eliminate?

People & LeadershipStrategy & ArchitectureCloud+57 more
CTO in Software, 201 - 500 employees
Without a doubt - Technical Debt! It's a ball and chain that creates an ever increasing drag on any organization, stifles innovation, and prevents transformation.
142 19 Replies
Read More Comments
40.8k views131 Upvotes319 Comments

Do you decide which projects have to be developed first based on the monetary benefits the business will receive or follow business priorities regardless of project costs? 

Process ManagementMobile DevelopmentSoftware Development
Read More Comments
838 views3 Comments