Report adoption, quality and the net time effect of an AI trial.

Instruction: Calculate the four requested measures in a compact exhibit, then give the recommendation you would say to the sponsor.

Context:

In this fictional one-month trial, 200 employees are eligible and licensed; 120 use the assistant at least once. The team completes 1,000 eligible tasks: 300 use AI and 240 of those AI-assisted tasks meet the quality threshold. For this simplified exercise, the 240 passing tasks each save six minutes against a measured matched-task baseline, already including ordinary review. The 60 failing tasks each require four extra minutes beyond the old workflow. Shared support takes ten additional hours. No other effects are measured. Calculate user adoption, task coverage, AI-task quality pass rate, and net hours saved. Treat the given differences as measured estimates, not universal performance claims.

Updated

Example Answer

I would report 60% monthly user adoption, 30% eligible-task coverage, an 80% pass rate among AI-assisted tasks, and ten estimated net hours saved. Those are different denominators, so I would show the calculation alongside each measure.

The trial has created a small positive capacity benefit after failures and shared support, under the supplied estimates. I would investigate the 60 failing tasks before arguing for greater coverage and measure whether released time has useful value. A rollout or purchase decision also needs costs and an acceptable quality standard, which this exhibit does not supply.

I would not call the ten hours cash savings or assume the result persists. Ordinary review is already included in the six-minute saving and must not be charged twice. Trying the tool once is not evidence of habitual use, and passing a quality threshold does not measure outcomes omitted from that threshold.

Worked dashboard exhibit

MeasureCalculationResult
Monthly user adoption120 users / 200 eligible employees60%
Eligible-task coverage300 AI-assisted tasks / 1,000 eligible tasks30%
AI-task quality pass rate240 passing AI-assisted tasks / 300 AI-assisted tasks80%
Passing-task time benefit240 × 6 minutes / 6024 hours
Failing-task extra work60 × 4 minutes / 604 hours
Shared supportGiven separately from task review10 hours
Net time effect24 − 4 − 1010 estimated hours saved

What the interviewer is assessing

Essential: Reports 60%, 30%, 80% and ten net hours, keeping employees separate from tasks and subtracting both the failing-task penalty and shared support.

Stronger: Recognizes that the time estimates already include ordinary review; ties any next spending decision to quality, financial costs and useful redeployment of capacity rather than adoption alone.

Red flags: Reports 24 hours as net benefit, subtracts review twice, calls the 80% pass rate employee adoption, or turns an hourly wage multiplier into realized payroll savings.

Follow-up

Only 60 of the 120 users return the following month. What rate would you report?

Example response: “Next-month retention of the original user cohort is 60 / 120, or 50%. I would separately report that month's adoption using its eligible population. New users belong in current adoption but not in the original cohort's retained-user numerator.”

Related Questions