Measuring AI code assistants and agents with the AI Measurement Framework
In this episode of Engineering Enablement, DX CTO Laura Tacho and CEO Abi Noda break down how to measure developer productivity in the age of AI using DX’s AI Measurement Framework. Drawing on research with industry leaders, vendors, and hundreds of organizations, they explain how to move beyond vendor hype and headlines to make data-driven decisions about AI adoption.
They cover why some fundamentals of productivity measurement remain constant, the pitfalls of over-relying on flawed metrics like acceptance rate, and how to track AI’s real impact across utilization, quality, and cost. The conversation also explores measuring agentic workflows, expanding the definition of “developer” to include new AI-enabled contributors, and avoiding second-order effects like technical debt and slowed PR throughput.
Whether you’re rolling out AI coding tools, experimenting with autonomous agents, or just trying to separate signal from noise, this episode offers a practical roadmap for understanding AI’s role in your organization—and ensuring it delivers sustainable, long-term gains.
Where to find Laura Tacho:
• LinkedIn: https://www.linkedin.com/in/lauratacho/
• LinkedIn: https://www.linkedin.com/in/abinoda
• Substack: https://substack.com/@abinoda
In this episode, we cover:
(00:00) Intro
(01:26) The challenge of measuring developer productivity in the AI age
(04:17) Measuring productivity in the AI era — what stays the same and what changes
(07:25) How to use DX’s AI Measurement Framework
(13:10) Measuring AI’s true impact from adoption rates to long-term quality and maintainability
(16:31) Why acceptance rate is flawed — and DX’s approach to tracking AI-authored code
(18:25) Three ways to gather measurement data
(21:55) How Google measures time savings and why self-reported data is misleading
(24:25) How to measure agentic workflows and a case for expanding the definition of developer
(28:50) A case for not overemphasizing AI’s role
(30:31) Measuring second-order effects
(32:26) Audience Q&A: applying metrics in practice
(36:45) Wrap up: best practices for rollout and communication
Referenced:
