How would you evaluate the performance of a recommendation system?

Instruction: Describe methods or metrics used to assess the effectiveness of a recommendation engine. Fictional fully judged candidate set: relevant items are {A,C,D}; the top-three ranking is [A,B,C]. Use binary relevance to compute precision@3, recall@3 and NDCG@3 with gain 1 for relevant items and discount 1/log2(rank+1). State how unknown relevance in real logs changes this interpretation.

Context: This question checks the candidate's knowledge on evaluating the success and relevance of recommendation systems, ensuring they know how to measure performance accurately.

Updated

Example Answer

I would separate offline ranking evaluation from live product impact. Offline, use a time-aware holdout and define relevance, the candidate set and top-k metrics such as recall or NDCG. Logged interactions only reveal feedback on exposed items; an unclicked or unseen item is not known to be irrelevant. Online, use an appropriately randomized experiment with a primary outcome and guardrails. I would also check latency, catalog coverage and important user/item slices instead of relying on one average.

Make it your own

Illustrative evaluation record: split cutoff, candidate universe, relevance rule, ranking cutoff, exposure logging, experiment assignment and latency/error guardrails.

Why this works

Offline relevance proxies and observed clicks have different limitations. Follow-up: how would sampled negative items change the interpretation of recall@k?

Sources: scikit-learn model evaluation, Offline contextual-bandit evaluation research, Microsoft trustworthy experiment design.

Worked practice case

Two of three recommended items are relevant: precision@3=2/3 and recall@3=2/3. DCG=1/log2(2)+0/log2(3)+1/log2(4)=1.5. IDCG=1+1/log2(3)+1/log2(4)≈2.13093, so NDCG≈0.70392. These calculations rely on the stipulated fully judged candidate set. In real exposure logs, unclicked or unshown items may have unknown relevance; treating every unobserved item as irrelevant changes the estimand and can reward the logging policy. Record candidate generation, judgments and cutoff alongside the score.

Related Questions