How do you choose between a larger model and a smaller routed model?

Instruction: Describe how you would decide whether to use one large model or route traffic across models of different sizes.

Context: Checks whether the candidate can explain the core concept clearly and connect it to real production decisions. Describe how you would decide whether to use one large model or route traffic across models of different sizes.

Updated

Example Answer

The way I'd approach it in an interview is this: I choose by comparing workflow value, not just offline accuracy. A larger model is often simpler operationally because it reduces routing complexity, but it may be too expensive or slow for the majority of traffic. A smaller routed model can be much more efficient if the router is good enough and the failure cases are well understood.

The key question is where the quality gap matters. If only a subset of requests truly needs the larger model, routing usually pays off. If routing errors create hard-to-diagnose regressions or fairness issues, one stronger model may be the better product choice.

I want the architecture that preserves reliability at the right cost, not the one that wins an abstract model-comparison debate.

Common Poor Answer

A weak answer is saying you should always use the smaller model when it is cheaper. Routing complexity and failure shape matter just as much as raw model price.

Your preparation path

Work through these questions in order. Read the answer aloud, then explain it in your own words.

1. Start with the foundations

Build the vocabulary and explain the core decisions.

  1. What drives latency in an LLM application besides the model itself? Free sample
  2. How would you explain batching to a product engineer? Free sample
  3. How would you explain throughput versus latency in model serving? Free sample

2. Apply it to a real workflow

Practice diagnosis, validation, and everyday tradeoffs.

  1. What signals matter most when optimizing cost without hiding quality loss? Free sample
  2. Traffic spikes create queueing even though the model service itself is healthy. How would you investigate? Member answer
  3. The product is fast on short prompts and unstable on long tool-using requests. What would you inspect? Member answer

3. Prepare for senior discussions

Explain failure boundaries, recovery, and production choices.

  1. How would you reason about tail latency in a multi-step LLM workflow? Member answer
  2. Long prompts stall active LLM streams. Would you separate prefill and decoding? Member answer
  3. How is KV-cache quantization different from weight quantization, and how would you validate it? Member answer

Related Questions