Experiment Readout & AI Summary | 2024-2025
💡 OVERVIEW
Designed an AI-generated summary module for Rokt's experiment readout experience, addressing a fundamental breakdown in how internal teams — Account Managers, Operations, and RPD — interpreted experiment data. Users landed on readout pages facing a wall of raw statistical output: parsing metrics, segment breakdowns, and significance thresholds independently, with no shared interpretive layer. Two people could read the same experiment and reach different conclusions. I owned this end to end, from problem framing through shipped experience.
✨ WHAT MADE IT MORE THAN A SUMMARY
What looked like a content feature was actually a design philosophy decision. The instinct would have been to improve the data visualization — better charts, cleaner tables. Instead, I reframed the problem: the readout page wasn't failing at presenting data, it was failing at generating understanding. The AI summary wasn't a shortcut around the data; it was a deliberate interpretive layer designed to meet three distinct user types where they actually were — Account Managers needing a quick client brief, Ops teams needing a clear status signal, RPD needing a fast strategic read. One format, three jobs. That tension shaped every decision about what the summary said and how it was structured.
🎯 IMPACT
The feature shipped and was adopted naturally — users gravitated toward the summary without being prompted, which is often the clearest signal of fit for an internal tool. Qualitative feedback was consistent across all three user groups: faster to read, easier to act on, more reliable than manual interpretation. Beyond adoption, this project marked a meaningful shift in how Rokt approached experimentation tooling — moving from a place to look at results to a place to understand them. That distinction became a reference point for how the team talked about the product going forward.