What drives latency in an LLM application besides the model itself?

Instruction: Explain the major sources of latency outside raw model inference time.

Context: Checks whether the candidate can explain the core concept clearly and connect it to real production decisions. Explain the major sources of latency outside raw model inference time.

Updated

Example Answer

I would trace the entire request from the client action to useful output. Prompt assembly, retrieval, authorization, queueing, tool calls, generation, serialization and rendering can all contribute. Add durations for sequential dependencies; parallel stages contribute through the critical path, so summing every span double-counts overlap.

For an illustrative request, 20 ms of setup precedes two independent authorized lookups taking 80 and 120 ms, then 300 ms of generation and 10 ms of rendering. If both lookups start together and both results are required, the idealized path is 20 + max(80, 120) + 300 + 10 = 450 ms. Running the lookups sequentially makes it 530 ms. Scheduling and network overhead can change those values in a real system.

I would distinguish first token, first useful output and completed response, then compare slow and typical request traces by workload. A healthy median model duration does not rule out queue or client problems. Stage budgets guide investigation, while end-to-end percentiles must be measured across requests rather than obtained by adding stage percentiles.

Common Poor Answer

Blaming model size alone misses the workflow. Adding every trace span also gives the wrong answer when work overlaps.

Serving timing metrics

Your preparation path

Work through these questions in order. Read the answer aloud, then explain it in your own words.

1. Start with the foundations

Build the vocabulary and explain the core decisions.

  1. What drives latency in an LLM application besides the model itself? Free sample
  2. How would you explain batching to a product engineer? Free sample
  3. How would you explain throughput versus latency in model serving? Free sample

2. Apply it to a real workflow

Practice diagnosis, validation, and everyday tradeoffs.

  1. What signals matter most when optimizing cost without hiding quality loss? Free sample
  2. Traffic spikes create queueing even though the model service itself is healthy. How would you investigate? Member answer
  3. The product is fast on short prompts and unstable on long tool-using requests. What would you inspect? Member answer

3. Prepare for senior discussions

Explain failure boundaries, recovery, and production choices.

  1. How would you reason about tail latency in a multi-step LLM workflow? Member answer
  2. Long prompts stall active LLM streams. Would you separate prefill and decoding? Member answer
  3. How is KV-cache quantization different from weight quantization, and how would you validate it? Member answer

Related Questions