What are the key metrics to evaluate the success of a recommendation system?

Instruction: Identify and explain the importance of different metrics in assessing a recommendation system's performance.

Context: This question probes the candidate's understanding of performance evaluation, focusing on how they measure effectiveness and the implications of various metrics.

Updated

Example Answer

I would define metrics with consistent units. Event-level click-through rate is attributed clicks divided by eligible logged impressions; a user-level click rate instead counts users in both numerator and denominator. Offline precision@k and recall@k require a stated relevance set and candidate universe. I would also measure ranked quality, catalog coverage, within-list diversity and operational latency. Online outcomes and satisfaction guardrails determine whether the model helps users; no single accuracy or engagement number settles that.

Make it your own

Illustrative arithmetic: 20 attributed clicks from 1,000 impressions gives 2% event CTR. If those clicks came from 10 of 200 exposed users, the user click rate is 5%, a different metric.

Why this works

The distinction prevents mixing people with recommendation events. Follow-up: how would you handle repeated clicks or a recommendation that was rendered but never visible?

Sources: Google CTR definition, Google recommendation re-ranking.

Related Questions