Speaker
Terran Melconian
Principal Applied Scientist - AI @Zillow, ML and Gen AI, Former Data Science Teacher & Engineering Director
Full speaker profileAbstract
Evaluation should not just produce metrics, which may turn out to be “metrics of convenience”, but should directly inform product and engineering decisions. Using examples from real products and their evaluations, this talk introduces a practical framework for decision-driven evaluation, both offline and online with real user testing. We will explore how to separate content quality from product impact, how to prioritize the riskiest uncertainties, and how to design evaluation systems that scale beyond spot checks and avoid whack-a-mole iteration.
The focus is on building confidence in when to ship, when to invest further, and when to pivot, not just improving scores.
Main Takeaways:
- Tie every evaluation to a decision.
- Separate content quality from product impact.
- Evaluate the riskiest uncertainty first.
- Build the right evaluation system for your product and stage.
Interview
Interview:
What is the focus of your work these days?
I work across a variety of ML projects, including propensity models, static prompt generative AI, and agentic experiences. I provide science guidance to a team of 25 scientists and engineers who develop and deliver Zillow’s market-leading products to real estate agents and mortgage professionals. I also cohost gen AI eval office hours, where teams from across the company come for reviews and advice on their Generate AI evaluation plans.
What is the motivation behind your talk?
I have seen many people implement evaluations by copying "best practices" from a podcast or a peer project, only to discover late in the process that the system they have built does not adequately inform the decision they need to make next. I want to help people start with the decision and use that to derive the most important properties their eval system needs to have.
Who is your session for?
This session is intended for senior engineers, applied scientists, and leaders working on machine learning and generative AI systems in production, or which are intended to go to production soon.