How would you investigate unequal pedestrian-detection error rates?

Instruction: In a hypothetical evaluation, group A has 10 misses among 100 labeled pedestrian encounters and group B has 40 misses among 200. Calculate both miss rates and the difference, then explain what additional evidence is needed before attributing the disparity or evaluating a mitigation.

Context:

Groups are supplied evaluation categories. The data does not establish legal categories, comparable scene conditions, causal mechanisms or fairness of a deployed system.

Updated

Worked answer

The observed miss rate is 10/100=10% for A and 40/200=20% for B. B's rate is 10 percentage points higher and twice A's rate. Comparing the raw miss counts alone would mix error rate with group size.

Inspect label quality, sampling, distance, lighting, occlusion, sensor configuration and other scene conditions. Report uncertainty and relevant intersectional slices; these counts alone do not establish why the rates differ. Matching or stratification may clarify specific comparisons, but it also changes the question being estimated and does not automatically prove causation.

Compare mitigation candidates on independent data, including both groups and downstream consequences. More diverse training examples, reweighting or preprocessing changes can help a diagnosed problem, but they may also affect calibration or another slice. Monitor false alerts as well as misses and avoid declaring success from an improved overall average.

Document the intended population, remaining gaps, stakeholder concerns and accountable release decision. Involve relevant expertise where fairness or legal obligations depend on context. Equal observed error rates alone would not establish that every relevant harm has been addressed.

Related Questions