Instruction: Describe how you would build an evaluation set for a support-facing RAG system.
Context: Assesses whether the candidate can design a practical architecture and explain the main tradeoffs. Describe how you would build an evaluation set for a support-facing RAG system.
Official answer available
Preview the opening of the answer, then unlock the full walkthrough.
I would build the eval set from real support work, not generic Q and A prompts. That means collecting historical tickets, help-center searches, escalation cases, and agent-authored resolutions, then turning them into labeled queries with gold evidence and expected handling.
I would explicitly include different classes: answerable...
easy
easy
easy
easy
easy
easy