How can multi-level bootstrap techniques improve the accuracy of confidence intervals in A/B tests with small sample sizes?
Instruction: Explain the technique and its benefits over traditional methods.
Updated
Example Answer
A multilevel bootstrap can help when observations are nested and treating every row as independent would understate uncertainty. I would start with the assignment and sampling design. If treatment is assigned to schools, students within a school are dependent, so a suitable cluster bootstrap resamples schools and keeps their student records together. Resampling individual students alone would act as though we had far more independent information.
This does not automatically solve a small-sample problem. Ten schools with a thousand students still provide only ten assignment clusters. With very few clusters, ordinary bootstrap confidence intervals can have poor coverage. Whether to resample within each selected cluster as well depends on how the data were sampled and the estimand. I would consider a design-appropriate randomization analysis or other small-cluster method, and check coverage in a simulation that resembles the experiment.
Why this works
It separates the reason for hierarchical resampling from a blanket claim of greater accuracy. The design determines which units are independent.
Follow-up to prepare for
How many independent clusters exist in each arm, and what variation does the proposed resampling scheme reproduce?