Has anyone had any success with evaluating the impact of using Generative AI tools such as GitHub's Copilot on the productivity or performance impact on developers? I see a lot of qualitative discussions about how developers say they are more productive, but how are you measuring that impact?

42.7k viewscircle icon55 Upvotescircle icon16 Comments
Sort by:
VP of IT in Manufacturing23 days ago

From my perspective, we proved the value of Generative AI (Development co-pilots) by focusing on two key areas. First, we measured our team's velocity, establishing a clear baseline before introducing the tool and seeing a sustained increase in story points completed per sprint afterward.
Second, we went beyond just counting pull requests. We tracked PR cycle time—the time from creation to merge—and saw a significant drop. For us, that was the key insight: we weren't just writing more code, we were delivering and merging it much faster without compromising quality.

Lightbulb on1
Global Practice Leader - ADM in IT Services24 days ago

There are multiple ways to check on the productivity improvements

Baseline Velocity / throughput of past X months which was without any code companion. Track the velocity / throughput post the code companion usage - to get a steady state view try for atleast 3 - 6 sprints : during this period ensure the developer is encourage to use the tool. The telemetry reports will provide how many are actively using the tool, how many prompt are being done / accepted etc. Make corrections based on this data report - if teams need more training provide so, if they need additional time to get used to using the tool provide so, we also have devised mechanisms to check lines of code generated - human vs machine generated (check latest announcements from GHCP for these aspects)

Once in regular use for build, you see the trend moving upwards & various quantitative & qualitative metrics of regular development will be able to show the outcomes - code quality, velocity, time to market etc

CEO in IT Servicesa month ago

I use GitHub Copilot almost daily, and I am an experienced Java developer. It actually makes me more productive when creating some patterns and refactoring, unit testing. I am exploring further at this point. What I can say is probably increasing my output 2x fold. The caveat is I still check the code generated for validity. I bet that senior devs will use the tool more efficiently due to the fact they know what to ask, based on their experience.

VP, Application Development in Finance (non-banking)a month ago

Any specific use cases that you might be able to share with using GitHub CoPilot? 

Lightbulb on1
Director of IT in Educationa month ago

Yes, these tools are very effective and efficient.

Content you might like

Yes33%

No – we have intent-based API monitoring but don’t use it for AI security 58%

We don’t use intent-based API monitoring8%

Other/unsure

View Results

Workvivo7%

Unily5%

Service Now39%

Interact36%

Sitecore18%

Simpplr23%

Igloo4%

Other (add comment)4%

View Results