Yegor Denisov-Blanch, a researcher at Stanford, presents the results of a large-scale study on the impact of AI on the productivity of more than 100,000 developers. The objective is to move past the "hype" and quantify the actual return on investment (ROI).
The study reveals a widening gap: the performance difference between teams that succeed with AI and those that fail is growing. Contrary to popular belief, the amount of AI used (number of tokens) correlates poorly with productivity. The determining factor is the quality of the code environment ("Codebase Hygiene"). Clean, well-tested, well-documented code allows AI to perform well. Conversely, on a degraded codebase, AI risks accelerating entropy and technical debt, requiring more human effort to correct errors.
Denisov-Blanch warns against simplistic metrics such as the number of Pull Requests (PRs). He cites the example of a company that observed a 14% increase in PRs, initially interpreted as a success. A more detailed analysis revealed a 9% drop in code quality and a 2.5x increase in "rework" (rework on recently written code). The gain in volume was offset by the drop in quality, making the ROI potentially negative.
To measure ROI correctly, Stanford proposes a framework: 1. Measure Usage: Distinguish theoretical access from actual usage (via fine-grained telemetry) and identify usage "patterns" (personal use vs. agentic orchestration). 2. Measure "Engineering Outcomes": Use a primary "Engineering Output" metric (based on an ML model trained to replicate human expert evaluation, rather than lines-of-code volume) coupled with "Guardrail" metrics (quality, rework rate, team health) that must be kept at a healthy level.
The conclusion is that AI is an amplifier: it accelerates both good and bad practices. Leaders must precisely measure impact to course-correct and invest in code hygiene to unlock AI's potential.