How would you measure whether an AI pilot actually improves analyst work?

Example Answer

I would agree on the eligible task types, a quality rubric, and a tolerable quality difference before the pilot. We would set the expansion rule before seeing results: evidence supporting at least 20% less active handling time while meeting that quality requirement, with serious errors reviewed separately. Active handling time would include drafting, checking, and corrections, because a fast draft that creates more rework is not an improvement.

I would randomly assign analysts within experience groups to assistant access or the current workflow, aiming for 20 in each group. Both groups would handle comparable assignments over the same four weeks. If analysts routinely share outputs, I would consider randomizing intact teams and account for the smaller number of independent groups.

Independent reviewers would score a sample of finished summaries without knowing which workflow produced them. I would track factual quality, serious errors, rework, and task completion alongside handling time. Usage and feedback would help explain whether access translated into adoption.

My primary comparison would include everyone assigned to each group, not just enthusiastic users. I would report the observed time difference, quality results, uncertainty, and important differences by task type. Multiple summaries from one analyst are correlated, so I would account for the analyst or assigned team in the analysis rather than treating every summary as an independent participant. Expansion requires evidence consistent with the agreed improvement and quality gate. Four weeks of sparse data may support another bounded pilot rather than a company-wide claim.

What the interviewer is assessing

Essential: Uses a concurrent randomized comparison, includes checking and corrections in handling time, specifies independent quality review, and analyzes analysts in their assigned groups even if some never use the assistant.

Stronger: Sets the improvement and quality decision rules in advance, accounts for analyst-level clustering and output sharing, and distinguishes usage diagnostics from causal evidence. Accept a carefully balanced crossover with randomized order if learning carryover and task comparability are addressed.

Red flags: Compares enthusiasts with unrelated historical averages, drops nonusers after assignment, treats every repeated task as an independent analyst, or declares success from draft speed alone.

Follow-up

What if frequent users improve but the assigned assistant group does not?

Example response: “I would investigate training, task fit, and access barriers. The frequent-user result is informative but can reflect selection bias. I would report the assigned-group outcome as primary and test an adoption improvement before claiming the tool works for everyone.”