What metrics would you use to assess the performance of an AI-driven recommendation system?

Example Answer

I would combine offline ranking checks with user outcomes and system guardrails. For a defined top-k recommendation list, precision@k measures the fraction of recommended items judged relevant; recall@k measures the fraction of the relevant set retrieved. State the judgment process and what happens when no relevant items are known. Historical non-clicks are not complete relevance labels.

For online measures, define the event and denominator consistently. Suppose 1,000 eligible recommendation impressions produce 80 recorded clicks and 12 attributed purchases within an agreed window. Impression CTR is 8%; purchase-per-click is 15%; purchase-per-impression is 1.2%. None is a percentage of unique users unless both numerator and denominator use unique users.

For retention, define a fixed eligible starting cohort, qualifying return action and window. If 120 of 200 original users return in that window after full follow-up, retention is 60%; later new users do not enter that denominator. Report missing observations.

Track useful discovery, complaints, diversity/coverage, latency and cost alongside business outcomes. Longer sessions may mean frustration and higher CTR may reflect clickbait. A randomized comparison with suitable guardrails can estimate incremental value; metric movement alone does not attribute success to the recommender.