QCon AI Boston 2026

Session

From AI Agent Demo to Production: Automated Testing and Evaluation

June 1–2, 2026 · Boston University

Monday Jun 1 / 01:20PM EDT · Metcalf Hall Large

Speaker

Zhou Yu

Founder @Arklex.ai and CS Professor at Columbia University - Co-Director of the DAPlab

Full speaker profile

Abstract

90% of the enterprise agents are stuck in POC. We don't know when the agents are good enough to deploy without a systematic testing and evaluation process. This presentation will share best practices, our practical dev tools, and lessons from the field to help you break out of the demo loop and build AI agents that deliver real business impact. In the talk, we will showcase an example of multi-turn agent simulation tool for automated agents

Main Takeaways:

  1. Simulation is critical for agent evaluation to reduce manual testing and improve agent quality pre-production and during iterations
  2. Multi-turn eval is necessary in agent eval to guarantee agent quality