Instruction: Describe how you would combine several kinds of graders into one evaluation system.
Context: Assesses whether the candidate can design a practical architecture and explain the main tradeoffs. Describe how you would combine several kinds of graders into one evaluation system.
Official answer available
Preview the opening of the answer, then unlock the full walkthrough.
I would put deterministic checks first for hard constraints like schema validity, forbidden actions, missing citations, or malformed tool calls. Those checks are cheap, legible, and should catch the obvious failures before anything probabilistic weighs in.
Then I would use model-based graders for scalable judgment on dimensions where deterministic checks...
easy
easy
easy
easy
easy
easy