Make a canary decision when the aggregate looks healthy.

Example Answer

The canary is failing at 1.8%, versus 0.1% for stable. The combined rate is only 0.134%, which hides the affected cohort. The canary has enough requests and exceeds the agreed 1% stop threshold, so I would halt promotion and route its traffic back to the compatible stable version.

I would verify recovery by version and endpoint, preserve the evidence, and investigate the changed code or configuration. The fixed rule is enough for this operational decision; I would not label it a universal statistical significance threshold.

Worked reasoning

The canary's rate is 18 times the stable rate. Its small traffic share dilutes the aggregate, so a fleet-wide chart can falsely reassure the operator.

Before reopening promotion, identify the failing paths, reproduce the trigger and test the fix. Compare latency and business correctness as well as errors. Retain compatible capacity while the routing change propagates and watch existing connections drain. A rollout job finishing is not the recovery criterion.

What the interviewer is assessing

Core checks (1 point each):

• Calculates 1.8%, 0.1% and 0.134% correctly.
• Applies the supplied cohort stop rule and compatible recovery path.
• Checks recovery and correctness beyond the aggregate dashboard.

Stronger (+1 after all core checks): Recognizes that canary sample representativeness and minimum exposure are part of the decision contract.

Acceptable alternatives: Disabling the identified feature can be acceptable if it is verified to restore the affected behavior faster and safely.

Red flags: Promotes because the aggregate is low or calls the arbitrary threshold statistical proof.

Follow-up

What if the canary received only internal test users?

Example response: The comparison would not represent the intended production population. I would still investigate observed failures, but a clean result could not justify general promotion. Route a bounded representative cohort or document the uncovered workloads before deciding.

Pressure test

A one-way data migration makes the old version incompatible. Do you still roll back?

Example response: No. Stop further exposure and use a compatible containment path, such as disabling the affected write feature or routing to a tested forward fix. Assess data already changed and reconcile it. The failure threshold still requires action, but it cannot authorize an incompatible binary rollback.

Scored practice examples

Each core point requires every part of its check. Missing or contradicted evidence earns no point for that check. Record unresolved unsafe actions separately; a total does not cancel them. The fourth point requires all three core checks plus the stronger criterion.

Weak — 0/4. The combined 0.134% is low, so promote the canary now. There is no need to calculate the separate cohorts or check recovery.

Assessment: Omits two required rates, violates the supplied stop policy and provides no recovery verification. Promoting through the known failing cohort is the unsafe gap. Core checks: 0/0/0. Unsafe gap: yes.

Adequate — 3/4. Canary is 36/2,000 = 1.8%; stable is 98/98,000 = 0.1%; combined is 134/100,000 = 0.134%. Canary exceeds 1% with at least 1,000 requests, so stop promotion and return traffic to the schema-compatible stable version. Verify errors and latency by version and endpoint, plus business correctness, before reopening promotion.

Assessment: All three core checks are evidenced without contradiction. The stronger criterion is not demonstrated. Core checks: 1/1/1. Unsafe gap: no.

Strong — 4/4. Canary is 36/2,000 = 1.8%; stable is 98/98,000 = 0.1%; combined is 134/100,000 = 0.134%. Canary exceeds 1% with at least 1,000 requests, so stop promotion and return traffic to the schema-compatible stable version. Verify errors and latency by version and endpoint, plus business correctness, before reopening promotion. The provided comparable request mix and 2,000 observations satisfy this operational policy, not a universal statistical significance claim. If the cohort contained only internal users, a clean result would not support general promotion; representative exposure would still be required.

Assessment: Connects representativeness and minimum exposure to the policy while rejecting an unsupported statistical claim. Core checks: 1/1/1; stronger: met. Unsafe gap: no.