How would you explain throughput versus latency in model serving?

Instruction: Describe the difference between throughput and latency in a serving system.

Context: Checks whether the candidate can explain the core concept clearly and connect it to real production decisions. Describe the difference between throughput and latency in a serving system.

Updated

Example Answer

Latency is the time a request takes; throughput is the amount of work completed per unit of time. In chat, I would separately measure time to first token, the gaps between subsequent outputs, and total completion time.

For example, completing 100 comparable jobs in 10 seconds is 10 jobs per second. That figure does not tell me whether some users waited eight seconds before seeing anything. It also becomes misleading if the new jobs are much shorter or failures are excluded.

I would benchmark with the same arrival pattern and prompt/output-length distribution, then report completed tasks or tokens per second alongside latency percentiles and error rates. Batching and higher concurrency can improve capacity, but can also increase waiting or memory pressure. The operating point should meet the workflow's latency and quality requirements, rather than maximize GPU utilization alone.

Worked example

Illustrative throughput: 100 completed jobs / 10 seconds = 10 jobs/second. Report the input/output lengths and failed or rejected jobs alongside that value.

Why this works

The small calculation is checkable and the answer explains why an aggregate capacity figure cannot replace request-level experience.

Follow-up to prepare for

Would you report requests per second or tokens per second? Use the unit that represents the workload and keep the workload distribution visible.

Your preparation path

Work through these questions in order. Read the answer aloud, then explain it in your own words.

1. Start with the foundations

Build the vocabulary and explain the core decisions.

  1. What drives latency in an LLM application besides the model itself? Free sample
  2. How would you explain batching to a product engineer? Free sample
  3. How would you explain throughput versus latency in model serving? Free sample

2. Apply it to a real workflow

Practice diagnosis, validation, and everyday tradeoffs.

  1. What signals matter most when optimizing cost without hiding quality loss? Free sample
  2. Traffic spikes create queueing even though the model service itself is healthy. How would you investigate? Member answer
  3. The product is fast on short prompts and unstable on long tool-using requests. What would you inspect? Member answer

3. Prepare for senior discussions

Explain failure boundaries, recovery, and production choices.

  1. How would you reason about tail latency in a multi-step LLM workflow? Member answer
  2. Long prompts stall active LLM streams. Would you separate prefill and decoding? Member answer
  3. How is KV-cache quantization different from weight quantization, and how would you validate it? Member answer

Related Questions