How would you measure answer quality when there is no single exact correct output?

Instruction: Describe how you would evaluate open-ended model outputs. Task: explain a supplied refund policy stating 'eligible within 30 days; review required after that.' Response A says 'All refunds are guaranteed within 60 days.' Response B says 'Within 30 days you are eligible; later requests need review.' Grade correctness and usefulness separately with 0=fail,1=partial,2=meets; justify each score.

Context: Checks whether the candidate can explain the core concept clearly and connect it to real production decisions. Describe how you would evaluate open-ended model outputs.

Updated

Example Answer

The way I'd approach it in an interview is this: When there is no exact target string, I judge the answer against the job it was supposed to do. That usually means a rubric around dimensions like factual support, completeness, actionability, tone, policy compliance, and whether the response moved the workflow forward.

I try to make those dimensions concrete enough that different reviewers can apply them consistently. For example, instead of asking whether an answer is "good," I ask whether it answered the user’s request, whether material claims were supported, whether it respected product boundaries, and whether it chose the right fallback when information was missing.

I also like pairwise comparisons for ambiguous outputs. Often it is easier and more stable to ask which of two answers better serves the task than to assign one absolute score. That tends to produce cleaner signals when multiple answers could all be acceptable.

Common Poor Answer

A weak answer is saying you would use BLEU, ROUGE, or exact-match style metrics for open-ended outputs. Those usually miss what makes the answer useful in product context.

Worked practice case

A gets correctness 0 because it contradicts both the window and guarantee; usefulness 0 because acting on it misleads the user about the requested policy. B gets correctness 2 and usefulness 2 for this bounded explanation task: it preserves the stated rule and is understandable. These scores do not claim that B can approve a particular customer's refund. Anchor the scale with the task context and allow acceptable paraphrases; exact string matching would be inappropriate. If the task additionally required filing a request, completeness criteria would change.

Your preparation path

Work through these questions in order. Read the answer aloud, then explain it in your own words.

1. Start with the foundations

Build the vocabulary and explain the core decisions.

  1. What problem should offline evals solve before you ship an AI feature? Free sample
  2. How would you explain the difference between task success and model quality? Free sample
  3. What makes a good regression set for an AI workflow? Free sample

2. Apply it to a real workflow

Practice diagnosis, validation, and everyday tradeoffs.

  1. What signals matter most when evaluating tool-using agents? Free sample
  2. Your tool-using agent passes answer-level evals and still makes bad tool calls. How would you fix the blind spot? Member answer
  3. Production feedback says answers feel less trustworthy even though offline metrics are flat. How would you investigate? Member answer

3. Prepare for senior discussions

Explain failure boundaries, recovery, and production choices.

  1. Human reviewers are expensive, but model graders miss subtle safety issues. How would you balance the two? Member answer
  2. Two coding agents have different scores and sandbox limits. How would you compare them fairly? Member answer
  3. An answer tells your LLM grader to award full marks. How would you keep the evaluation trustworthy? Member answer

Related Questions