How would you evaluate these model candidates for an internal assistant?

Instruction: Compare the candidates against the supplied requirements, recommend the next investment, and explain the evidence needed for the decision.

Context:

A fictional internal assistant has three mandatory pilot gates: at least 93% acceptable answers on a 200-case evaluation, response latency at the 95th percentile no greater than 2 seconds, and model cost no greater than $0.02 per task. Candidate A scores 92%, 1.5 seconds, and $0.025; B scores 94%, 3.2 seconds, and $0.018; C scores 90%, 1.9 seconds, and $0.012. All three pass the initial data-handling review. Figures are invented, and the evaluation uses the same inputs and scoring rubric.

Updated

Official answer available

Read the opening below, then unlock the full answer and practical guidance.

I would not declare a winner because none currently meets the agreed gates. A misses quality and cost, B misses latency, and C misses quality. A weighted average would conceal those failures unless the sponsor explicitly changed the requirements. My first investigation would be B because it meets the observed quality and model-cost gates...

Upgrade to view official answer

Related Questions