Report release-test status without hiding blocked work or retries.

Instruction: Calculate distinct-case execution and outcome metrics, explain their denominators and give a concise dashboard statement.

Context:

Fictional case: a release plan has 200 distinct required test cases. At the snapshot, 150 have a final passing outcome, 10 have a final failing outcome, 25 are blocked before any execution, and 15 have not started. Of the 150 final passes, 12 initially failed and passed on exactly one retry. Each of the 10 final failures was executed once with no retry. No other retries occurred, and no cases were removed from the plan. Report final pass rate among executed cases, final pass share of the full plan, execution coverage, blocked share and first-attempt pass rate. A retry creates another attempt, not another required case.

Updated

Example Answer

We have executed 160 of 200 required cases, or 80%. Of those executed cases, 150 finally pass, so the executed-case final pass rate is 93.75%. Final passes cover 75% of the full plan. The 25 blocked cases are 12.5% of the plan, and 15 cases, or 7.5%, have not started.

Twelve final passes needed a retry. The first-attempt pass count is therefore 138, or 86.25% of the 160 executed distinct cases. There were 172 attempts in total, but I would not use that as the denominator for distinct-case coverage.

My dashboard would show these counts together with blocked-case risks and owners. A 93.75% final pass rate alone hides both untested work and retry instability; none of these aggregate numbers alone decides release.

Dashboard snapshot

MetricCalculationResult
Executed distinct cases(150 + 10) / 200160 cases; 80% of plan
Final pass rate among executed150 / 16093.75%
Final pass share of plan150 / 20075%
Blocked / not started25 / 200; 15 / 20012.5% blocked; 7.5% not started
First-attempt pass rate(150 − 12) / 160138 cases; 86.25%
Attempts and retry recoveries160 + 12 attempts; 12 / 160 cases172 attempts; 7.5% of executed cases passed after retry

What the interviewer is assessing

Core checks (1 point each):

  • Computes 160/200 execution and 150/160 final executed pass rate.
  • Reports 150/200 full-plan passes, 25 blocked and 15 unstarted.
  • Computes 138/160 first-attempt passes without treating retries as new cases.

Stronger (+1 after all core checks): States both counts and rates, distinguishes blocked from failed and not-started, and asks which risks the remaining cases represent before interpreting release readiness.

Acceptable alternatives: Other dashboards are valid if denominators and blocked work remain explicit.

Red flags: Calls blocked cases passing, reports 150/172 as the required-case pass rate, calls 200 cases executed, or infers release readiness from a single percentage.

Follow-up

A manager asks you to remove the 25 blocked cases so the report looks healthier. What do you show?

Example response: “I would retain the approved 200-case baseline and show blocked cases with causes, owners and impact. If scope is legitimately changed, I would version the plan, obtain the proper risk decision and show the removed scope explicitly rather than rewriting the old denominator. A presentation change must not conceal missing evidence.”

Related Questions