Design a testing strategy for tool contracts used by agents.

Instruction: Combine deterministic contract checks with task-based model evaluation and injected execution failures. Old tool results were {ok:true} only after completion. New results are {status:"pending",operation_id:"X"} or {status:"completed",operation_id:"X"}. An old parser treats any nonempty object as success. Give deterministic and model-level tests, including a timeout after remote completion.

Context: Test tools used by agents for both correct API behavior and correct tool selection under ambiguous user requests.

Updated

Official answer available

Read the opening below, then unlock the full answer and practical guidance.

I would keep three test layers. Contract tests validate required fields, types, output shape, and documented errors. Integration tests exercise authorization, business rules, and downstream failures. Task evaluations check whether the assistant selects the right tool, uses the right arguments, and reports the actual outcome.

For a hypothetical ticket...

Upgrade to view official answer

Your preparation path

Work through these questions in order. Read the answer aloud, then explain it in your own words.

1. Start with the foundations

Build the vocabulary and explain the core decisions.

  1. When should a model call a tool instead of answering from context? Free sample
  2. How would you explain MCP to a product engineer in one minute? Free sample
  3. What information should the orchestrator validate before executing a tool call? Free sample

2. Apply it to a real workflow

Practice diagnosis, validation, and everyday tradeoffs.

  1. Why do idempotency and retries matter more with AI-driven tool use? Free sample
  2. How do you keep permission boundaries clear when many tools share overlapping capabilities? Member answer
  3. A user revokes access to a connected app mid-session. How should the AI workflow behave? Member answer

3. Prepare for senior discussions

Explain failure boundaries, recovery, and production choices.

  1. Design a fallback strategy when tools are unavailable, degraded, or unauthorized. Member answer
  2. How do you implement MCP step-up authorization without losing scopes or retrying forever? Member answer
  3. An MCP request resumes after user confirmation. How would you validate its requestState? Member answer

Related Questions