Design a latency budget for a multi-step agent workflow.

Instruction: Describe how you would budget latency across planning, retrieval, tool use, and generation. Measured single-request trace: intake 0.1 s, retrieval 0.4 s, planning 0.6 s, two required tools in parallel taking 0.8 s and 1.2 s, then generation 0.5 s. Overall deadline is 3.0 s with 0.2 s delivery reserve. Compute critical-path duration and slack; distinguish this from adding stage p95 values.

Context: Assesses whether the candidate can design a practical architecture and explain the main tradeoffs. Describe how you would budget latency across planning, retrieval, tool use, and generation.

Updated

Official answer available

Read the opening below, then unlock the full answer and practical guidance.

I would work backward from the user promise and give the request an overall deadline. Intake, retrieval, model turns, tools, and final delivery need budgets within that deadline, including room for a useful fallback...

Upgrade to view official answer

Your preparation path

Work through these questions in order. Read the answer aloud, then explain it in your own words.

1. Start with the foundations

Build the vocabulary and explain the core decisions.

  1. What drives latency in an LLM application besides the model itself? Free sample
  2. How would you explain batching to a product engineer? Free sample
  3. How would you explain throughput versus latency in model serving? Free sample

2. Apply it to a real workflow

Practice diagnosis, validation, and everyday tradeoffs.

  1. What signals matter most when optimizing cost without hiding quality loss? Free sample
  2. Traffic spikes create queueing even though the model service itself is healthy. How would you investigate? Member answer
  3. The product is fast on short prompts and unstable on long tool-using requests. What would you inspect? Member answer

3. Prepare for senior discussions

Explain failure boundaries, recovery, and production choices.

  1. How would you reason about tail latency in a multi-step LLM workflow? Member answer
  2. Long prompts stall active LLM streams. Would you separate prefill and decoding? Member answer
  3. How is KV-cache quantization different from weight quantization, and how would you validate it? Member answer

Related Questions