Instruction: Explain what you would monitor once a tool-using workflow is in production.
Context: Assesses whether the candidate can design a practical architecture and explain the main tradeoffs. Explain what you would monitor once a tool-using workflow is in production.
Official answer available
Preview the opening of the answer, then unlock the full walkthrough.
I would monitor the workflow at both outcome level and step level. Outcome metrics tell me whether users are completing the job. Step metrics tell me where the workflow is drifting. For a tool-using system, I want visibility into tool selection, argument validity, retries, timeouts,...
easy
easy
easy
easy
easy
easy