How do you keep eval sets from drifting away from real user traffic?

Example Answer

The way I'd approach it in an interview is this: I keep evals aligned by treating production traffic as the source material, not as an afterthought. That means sampling real traces, support tickets, thumbs-down feedback, and incident reviews on a regular cadence and using them to refresh slices in the benchmark.

I also compare eval mix to traffic mix. If the product has shifted toward multi-turn troubleshooting or multilingual usage, but the benchmark still looks like last quarter’s FAQ assistant, the suite is lying to us politely.

The key is controlled refresh, not constant churn. I want a stable core set for regression protection and a rotating layer that reflects new behavior, new customers, and new failure patterns. That gives me both comparability and relevance.

Common Poor Answer

A weak answer is saying you update the eval set whenever something embarrassing happens. That creates recency bias and does not guarantee alignment with actual traffic distribution.