Talk
Evaluating the AI-Assisted Developer Experience
The rapid adoption of AI-assisted coding tools has fundamentally changed how developers approach complex engineering tasks. As the industry shifts toward agentic workflows, the visibility gap into how these tools impact overall developer productivity and code quality has widened. Engineering teams are now tasked with ensuring their developer experiences are reliable, accurate, and trustworthy, yet traditional testing methods often fall short in these indeterminate, AI-driven environments. This talk explores a framework for implementing developer experience testing at scale. We will examine how to build automated evaluation platforms that function as continuous integration for AI systems. Instead of relying on anecdotal feedback, we will discuss how to define and measure success using metrics that go beyond simple functionality. By the end of this session, attendees will walk away with two primary insights: How to create practical, everyday measurements that clearly show if AI tools are helping developers succeed or creating new hurdles. A straightforward strategy for monitoring AI performance over time, ensuring your tools remain helpful partners rather than sources of frustration. Attendees will leave with a clear plan to protect developer trust while embracing new ways of working, ensuring that the software they build remains high-quality and reliable, no matter how much automation is involved.
