This in-depth analysis explores how 18 major tech companies, including Google, GitHub, Microsoft, and Dropbox, measure the impact of AI on software development, amid the challenge of justifying growing investments in AI coding tools. Written by Gergely Orosz and Laura Tacho (CTO of DX), the article notes that while 85% of engineers use AI tools, many engineering leaders struggle to assess their real value, lacking clear metrics beyond superficial measures such as lines of code (LOC).
Central message: combine metrics
Effectively measuring AI impact requires combining existing 'core' engineering metrics with new AI-specific metrics. Companies should not abandon traditional metrics such as Change Failure Rate, PR throughput, PR cycle time, and developer experience, since the ultimate goal of AI is precisely to improve these software delivery fundamentals. These core metrics must be tracked alongside AI adoption rates, satisfaction (CSAT) with the tools, time saved per engineer, and AI spend. Dropbox, for example, reached 90% AI adoption and saw its engineers merge 20% more pull requests with a reduced change failure rate.
Segmentation and an experimental mindset
A crucial aspect is breaking down metrics by level of AI usage: comparing AI users to non-AI users, and analyzing trends over time. This breakdown by role, seniority, or programming language helps identify which groups benefit most from AI or need additional training. The article emphasizes an experimental mindset, where data is used to answer specific questions and test predictions about AI's influence.
Quality, maintainability, developer experience
Vigilance over code quality, maintainability, and developer experience is paramount. The authors warn that AI-assisted development can create "the biggest pile of technical debt" if not managed carefully. It is essential to track metrics that check each other, such as speed alongside quality (PR throughput and CFR). Beyond system metrics, self-reported data on "confidence in changes," "code maintainability," and "perceived quality" are vital for capturing long-term impacts. Developer experience, often wrongly reduced to superficial perks, is critical for reducing friction across the entire development cycle.
Emerging trends and challenges
Microsoft uses "bad developer days" (BDD) to assess AI's impact on daily friction, while Glassdoor measures experimentation outcomes (A/B tests). The acceptance rate of AI suggestions, once a benchmark metric, is declining because it is too narrow: it captures neither maintainability, nor bug introduction, nor overall productivity. Cost analysis, still rarely practiced so as not to discourage usage, is expected to receive greater scrutiny as AI budgets grow. Agent telemetry and measurement beyond code writing are identified as areas set to evolve significantly.
AI Measurement Framework and data layers
The article introduces the AI Measurement Framework, a recommended set of metrics blending AI metrics with core engineering metrics, with developer experience at its center. It advocates layered data collection: quantitative system data (AI tools, GitHub, JIRA, CI/CD), periodic qualitative surveys, and in-the-moment experience sampling. Monzo Bank's experience serves as a case study: objective measurement is difficult (data retention by vendors), but engineers' subjective sentiment and specific use cases such as code migrations demonstrate clear value.