How do you decide which requests deserve the expensive model?

Instruction: Explain how you would choose when to route traffic to a higher-cost model.

Context: Checks whether the candidate can explain the core concept clearly and connect it to real production decisions. Explain how you would choose when to route traffic to a higher-cost model.

Updated

Example Answer

The way I'd approach it in an interview is this: I decide based on expected value of extra quality. Requests that are ambiguous, high-stakes, multi-step, or historically error-prone are the best candidates for the expensive model. Straightforward or repetitive requests often do fine on the cheaper path.

I also want the routing rule to be observable. If the expensive model is being used, I should be able to explain why. That makes cost control and fairness review much easier later.

The wrong way to do this is to route everything that feels important in the moment. The right way is to use measurable signals tied to quality lift.

What I always try to avoid is giving a process answer that sounds clean in theory but falls apart once the data, users, or production constraints get messy.

Common Poor Answer

A weak answer is saying complex requests should use the expensive model. The better answer defines how complexity is detected and whether the extra model actually changes outcomes.

Your preparation path

Work through these questions in order. Read the answer aloud, then explain it in your own words.

1. Start with the foundations

Build the vocabulary and explain the core decisions.

  1. What drives latency in an LLM application besides the model itself? Free sample
  2. How would you explain batching to a product engineer? Free sample
  3. How would you explain throughput versus latency in model serving? Free sample

2. Apply it to a real workflow

Practice diagnosis, validation, and everyday tradeoffs.

  1. What signals matter most when optimizing cost without hiding quality loss? Free sample
  2. Traffic spikes create queueing even though the model service itself is healthy. How would you investigate? Member answer
  3. The product is fast on short prompts and unstable on long tool-using requests. What would you inspect? Member answer

3. Prepare for senior discussions

Explain failure boundaries, recovery, and production choices.

  1. How would you reason about tail latency in a multi-step LLM workflow? Member answer
  2. Long prompts stall active LLM streams. Would you separate prefill and decoding? Member answer
  3. How is KV-cache quantization different from weight quantization, and how would you validate it? Member answer

Related Questions