As Principal PM for the Enterprise Adoption team, I shaped a bulk-testing feature that would allow every brand to test their AI Agent against custom evaluation criteria for all of their key use cases before launch — now used for every implementation.
AI Managers had no safe way to know what a change would do before it hit real customers — testing was manual, reactive, and lived in staff-only tools. As Principal PM, I defined the strategy, requirements, and vision for a testing platform that lets customers simulate and evaluate AI Agent conversations at scale.
Testing generative AI isn't like testing deterministic software — the same input can produce different, equally valid outputs. The heart of this project was designing an evaluation framework where customers define quality on their own terms: natural-language success criteria, judged Pass/Fail by an LLM, held to a measurable bar of agreement with human annotators. Test cases became the goalposts that define what a "good" AI Agent meant.
Everyone agreed testing mattered; nobody agreed on where to start. Simulation fidelity, version control, drafts, deployments — the problem space sprawled. I ran prework alignment, engineering experiments, and UX working sessions to converge fourteen opinionated people (plus four Director+ stakeholders) on a narrow, extensible foundation: single-turn simulation first, evaluation second, dashboard testing third, with other iterations deliberately deferred.
The cheapest version of this feature would have been a bolted-on testing tool. Instead, I aligned the team on building simulation and evaluation as reusable domains — foundations that other product teams could build on, that could absorb Ada's fragmented internal tooling, and that could one day ship as standalone APIs for evaluating any customer service transcript, human or AI.
.png){kind=small}
