Which triage model would you recommend when accuracy and error cost disagree?

Question

A fictional operations queue contains 1,000 cases per day: 100 are truly urgent and 900 are not. Model A flags 80 urgent and 90 nonurgent cases; it misses 20 urgent cases and correctly leaves 810 nonurgent cases unflagged. Model B flags 95 urgent and 180 nonurgent cases; it misses 5 urgent cases and correctly leaves 720 nonurgent cases unflagged. Every flagged case receives timely human review. Capacity is 300 reviews per day. For this exercise, each missed urgent case causes an expected $500 loss and each unnecessary review costs $10. Correct classifications have zero incremental cost. The cases and losses are fictional; use the same daily cohort for both models.

Accepted Answer

Example Answer

I recommend Model B for the stated queue. It catches more urgent cases, fits the 300-review daily capacity, and reduces the modeled daily error cost from $10,900 to $4,300. Its lower overall accuracy is acceptable here because urgent misses are much more costly than unnecessary reviews.

B requires 275 reviews a day, leaving 25 slots of daily headroom. I would authorize deployment at this operating point with the review staffing and queue monitoring included in the plan. The supplied promise of timely review is essential: an hourly backlog or ineffective review could change the result. I would monitor urgent misses, queue delay, and reviewer outcomes, and check the stated loss assumptions against actual incidents.

Worked exhibit

Measure	Model A	Model B
Correct classifications	80 + 810 = 890	95 + 720 = 815
Overall accuracy	890 / 1,000 = 89%	815 / 1,000 = 81.5%
Urgent-case recall	80 / 100 = 80%	95 / 100 = 95%
Daily reviews	80 + 90 = 170	95 + 180 = 275
Daily review headroom	300 − 170 = 130	300 − 275 = 25
Missed-urgent loss	20 × $500 = $10,000	5 × $500 = $2,500
Unnecessary-review cost	90 × $10 = $900	180 × $10 = $1,800
Total stated daily error cost	$10,900	$4,300

B avoids 15 urgent misses at $500 each ($7,500) and adds 90 unnecessary reviews at $10 each ($900): net modeled benefit $6,600 per day. Correct classifications incur zero incremental cost under the exercise assumptions; necessary urgent reviews are not assigned an extra cost.

What the interviewer is assessing

Essential: Calculates accuracy as 89% versus 81.5%, urgent recall as 80% versus 95%, review volumes as 170 versus 275, and daily error costs as $10,900 versus $4,300; recommends B under the supplied assumptions.

Stronger: Connects the $6,600 difference to 15 avoided urgent misses and 90 extra unnecessary reviews, preserves the denominator of 100 urgent cases, and distinguishes daily headroom from peak handling. A different choice requires explicitly changing an assumption or adding a real constraint.

Red flags: Chooses A solely for accuracy, counts every review as unnecessary, invents results at a new threshold, or claims the cost comparison proves the model safe.

Follow-up

How much would an urgent miss need to cost for the models to tie?

Example response: Let an urgent miss cost x. A costs 20x + $900; B costs 5x + $1,800. Equality gives 15x = $900, so x = $60. Above $60 B has lower modeled error cost; below it A does. This comparison keeps capacity, review effectiveness, and all other assumptions unchanged.

Pressure test

Changed constraint: Review capacity falls to 200 cases per day.

Adaptive response: B exceeds capacity by 75 reviews a day. My first choice is to secure 75 additional timely slots and compare their incremental cost with B’s $6,600 daily advantage, including the difficulty of staffing peak demand. If extra capacity is unavailable, I would evaluate a different threshold or routing policy before deploying B; this table cannot predict those results. A fits the new capacity but produces 15 additional urgent misses. The accountable owner must explicitly decide whether that loss is acceptable relative to the existing workflow; I would not let 75 unreviewed cases quietly accumulate.