How do you decide whether reranking is worth the latency?

Example Answer

The way I'd approach it in an interview is this: I decide reranking the same way I decide any extra serving stage: does it materially improve the top few candidates on the queries that matter most? If first-stage retrieval already brings the right evidence to the top, reranking is just latency. If the right material is present but misordered often enough to hurt answers, reranking is usually worth testing.

I look specifically at near-miss behavior. Are we getting the right chunk at rank 8 instead of rank 1? Are duplicates crowding out better passages? Are exact-match chunks beating more answer-bearing semantic ones? Those are classic reranker wins.

I also do not assume reranking must run on every request. Often the best design is selective: use it for high-value workflows, ambiguous queries, or large candidate sets. That captures most of the quality gain without paying the latency tax everywhere.

Common Poor Answer

A weak answer is, "If quality matters, I would always add a reranker." That is not a decision rule. It ignores where the current ranker is already good enough and where reranking actually moves the top candidates.