How do you choose between human review and model graders?

Example Answer

The way I'd approach it in an interview is this: I choose based on the kind of judgment required and the cost of getting it wrong. If the task needs nuanced safety judgment, domain expertise, or interpretation of ambiguous edge cases, I want humans in the loop at least for calibration and critical slices. If the task is repetitive, well-rubriced, and high-volume, model graders can buy a lot of scale.

The mistake is treating this like a binary decision. In practice, the best system is layered. Deterministic checks catch hard failures cheaply. Model graders handle broad coverage. Humans review the slices where subtlety, severity, or uncertainty are highest.

I also continuously compare the graders to humans. A grader that was acceptable last month can drift as prompts, models, and product behavior change. If I stop calibrating, I end up trusting automation more than the evidence justifies.

Common Poor Answer

A weak answer is saying model graders replace humans once the rubric is good enough. For high-stakes or subtle judgments, that confidence is usually premature.