Design a rubric for judging grounded answers in a RAG assistant.

Instruction: Describe how you would build a grading rubric for a retrieval-based assistant. Source S1: 'Plan Basic retains exports for 7 days.' Answer A: 'Basic exports last 7 days [S1], and deleted exports can always be restored [S1].' Answer B: 'Basic exports last 7 days [S1]. The supplied source does not establish recovery after deletion.' Grade support, citation correctness and completeness for a user who asked retention and recovery.

Context: Assesses whether the candidate can design a practical architecture and explain the main tradeoffs. Describe how you would build a grading rubric for a retrieval-based assistant.

Updated

Official answer available

Read the opening below, then unlock the full answer and practical guidance.

I would start the rubric with the core question: did the answer stay inside what the retrieved evidence can actually support...

Upgrade to view official answer

Your preparation path

Work through these questions in order. Read the answer aloud, then explain it in your own words.

1. Start with the foundations

Build the vocabulary and explain the core decisions.

  1. What problem should offline evals solve before you ship an AI feature? Free sample
  2. How would you explain the difference between task success and model quality? Free sample
  3. What makes a good regression set for an AI workflow? Free sample

2. Apply it to a real workflow

Practice diagnosis, validation, and everyday tradeoffs.

  1. What signals matter most when evaluating tool-using agents? Free sample
  2. Your tool-using agent passes answer-level evals and still makes bad tool calls. How would you fix the blind spot? Member answer
  3. Production feedback says answers feel less trustworthy even though offline metrics are flat. How would you investigate? Member answer

3. Prepare for senior discussions

Explain failure boundaries, recovery, and production choices.

  1. Human reviewers are expensive, but model graders miss subtle safety issues. How would you balance the two? Member answer
  2. Two coding agents have different scores and sandbox limits. How would you compare them fairly? Member answer
  3. An answer tells your LLM grader to award full marks. How would you keep the evaluation trustworthy? Member answer

Related Questions