Investigate a release-time latency change.

Instruction: Rank two hypotheses, choose the next two observations, and propose containment with a recovery check.

Context:

A fictional API rollout moves 30% of traffic to v2. Both cohorts receive comparable endpoints and request sizes; requests per second are unchanged. Values below cover the same five-minute interval. The traces are selected slow requests, not a representative latency distribution. The prior binary remains schema-compatible.

Observationv1 cohortv2 cohort
Request p95220 ms1,800 ms
CPU34%35%
Server error rate0.05%0.06%
Pool checked out / maximum, at snapshot8 / 2020 / 20
Pool wait in one slow trace5 ms1,400 ms
SQL execution in that same trace78 ms80 ms

No connection-lifetime, transaction-age or database-capacity data has been supplied. Identify what would distinguish a v2 connection-lifecycle problem from slow or blocked database work.

Updated

Example Answer

The release comparison points first to connection acquisition: v2 has 1.4 seconds of pool wait while the sampled SQL is about as fast as v1. I would compare checkout lifetimes and connection returns by version, then inspect database session state, transaction age and blockers. Neither a full pool nor quiet CPU proves a connection leak.

I would pause promotion and route the affected cohort back to the compatible v1 while checking capacity. Recovery requires latency and pool waits to improve under comparable traffic. Increasing every pool without database evidence could amplify the failure.

Worked reasoning

HypothesisNext evidenceWhat weakens it
v2 retains connections too longCheckout duration and return counts, linked to request completion/error pathsConnections return promptly while database work remains blocked
Long or blocked transactions occupy the poolSession/transaction age, wait events and blockers for those connectionsServer work completes promptly but completed requests still retain checkouts

The first observation follows the application lifecycle; the second follows the same connections on the server. A transaction may remain open while its session is idle, so query execution time alone cannot exclude database-side retention. Compare more traces and endpoint cohorts before generalizing from the examples.

For containment, stop promotion, verify v1 headroom and move exposure in bounded steps. Watch request p95/p99, successful completions, connection wait and database saturation. If rollback clears waits but not all latency, keep investigating the remaining spans; containment is not a complete root-cause explanation.

Changed-constraint alternative: Suppose a completed schema change makes v1 incompatible, but a separately verified v2 flag bypasses the new optional enrichment path. Disable that path and bound admission while keeping v2. Verify that required API behavior remains correct and database occupancy falls. The changed fact removes binary rollback; the invariant is compatible containment with measured user recovery. Without a verified bypass, choose explicit degraded service rather than inventing one.

What the interviewer is assessing

Core checks (1 point each):

  • Uses the cohort and span evidence to prioritize connection acquisition without declaring a leak proved.
  • Names application checkout-lifetime evidence and database transaction/wait evidence that distinguish the hypotheses.
  • Chooses compatible, capacity-aware containment and verifies both user latency and pool behavior.

Stronger (+1 after all core checks): Explains how a short SQL span can coexist with an idle open transaction or an unreleased application checkout.

Acceptable alternatives: A verified feature disable or admission limit can replace rollback when its compatibility and user impact are explained.

Red flags: Treats low CPU as proof of spare end-to-end capacity, declares every database query healthy from one trace, or raises pools without checking database capacity.

Follow-up

Rollback reduces pool waits but latency remains 900 ms. Are you done?

Example response: No. Split the remaining latency by endpoint and span, check queued work draining and downstream response time, and continue impact reporting. The change supports a release-related component; it does not establish full recovery or a single cause.

Related Questions