Imagine a scenario where an AI product you manage is facing user adoption challenges despite strong technical performance. How would you approach this situation?

Instruction: Interpret the fictional adoption funnel, identify evidence needed to distinguish causes, and propose a targeted improvement and success measure.

Context:

An AI feature has strong results on its technical benchmark but low use. In a fictional cohort of 1,000 eligible users, 200 tried it, 120 of those completed the intended task once, and 30 of those completers repeated the task. All users had the same observation window and follow-up opportunity; the groups are nested. Completion is instrumented but output acceptance and reasons for non-use have not been measured.

Calculate the trial, first-completion and repeat rates with clear denominators. Explain what you would investigate before choosing onboarding, workflow or model changes.

Updated

Example answer

The trial rate is 200/1,000 = 20%. First completion among triers is 120/200 = 60%. Repeat use among first completers is 30/120 = 25%, and repeat use across the eligible cohort is 30/1,000 = 3%. These denominators describe different stages. Platform logins would not tell us whether people used this feature or completed its intended task.

I would verify the events, eligibility and observation window, then segment by the workflow and relevant user needs. Strong benchmark performance may not represent actual inputs, latency, output acceptance or the cost of checking results. Completion also does not establish that the output was useful.

Talk with people who never tried, abandoned, completed once and returned. Observe representative tasks to distinguish discovery or access problems, confusing interaction, integration friction, insufficient trust and lack of repeat need. A one-time task can have low repeat use without being unsuccessful; choose the outcome based on the intended use frequency.

The intervention should follow the finding. If eligible users cannot discover the feature, test an appropriate entry point. If they abandon while correcting outputs, improve that workflow or quality before adding promotional prompts. If the product solves no important repeated need, reconsider the use case rather than optimize a misleading retention target.

For a chosen change, run an appropriate controlled experiment with eligible cohorts and a defined window. Measure accepted task completion or end-to-end effort, relevant repeat use, errors and support burden. Keep accessibility and user control in the design. A before-and-after increase in clicks alone would not show that the change caused lasting value.

Related Questions