Developing a Comprehensive Strategy for Estimating the Causal Impact of a Software Release on System Performance
Instruction: Outline a detailed strategy for assessing the causal impact of a new software release on system performance, considering potential confounders.
Context: This question assesses the candidate's ability to design a causal inference study in a tech environment. Candidates should discuss methods to isolate the effect of the software release from other variables that may affect system performance, such as hardware changes or simultaneous software updates.
Updated
Official answer available
Read the opening below, then unlock the full answer and practical guidance.
Define the release, eligible workload, assignment unit and performance estimand. Distinguish a request-weighted mean, a session-weighted mean, tail latency, error rate and resource cost; they answer different questions. State how timeouts and failed requests enter the outcome.
Prefer a randomized rollout when feasible, with stable assignment and equivalent traffic eligibility...