What metrics would you consider when testing the usability of a website using A/B testing?

Instruction: Identify and justify the choice of metrics that would be important when assessing website usability through A/B testing.

Context: This question tests the candidate's understanding of usability metrics and their ability to select appropriate measures for evaluating website usability in an A/B testing framework.

Updated

Official answer

Choose metrics tied to a defined user task. For a checkout usability change, a primary outcome could be successful completion within a fixed window per assigned eligible user. Compare concurrent randomized variants, using consistent event definitions and attribution.

Useful secondary measures include error/retry rates, abandonment, support contacts and time to complete the task. Define what happens when a user does not finish: comparing time only among completers can favor a variant that drives slower users away. Report completion and time together or use a justified method that handles incomplete tasks.

More page views or time on site can mean interest, confusion or extra effort. Lower bounce can also reflect instrumentation or navigation changes. Interpret those measures in context rather than treating a direction as inherently better.

Use task-focused satisfaction questions and qualitative sessions to understand problems; NPS measures recommendation intent and is not a direct usability score. Include accessibility, latency and error guardrails. Prespecify the main decision metric and handle multiple comparisons; a convenient dashboard of many favorable metrics is not a reliable success criterion.

Related Questions