How would you evaluate multiple predicted trajectories for an animal near a road?
Instruction: A predictor returns several plausible animal trajectories. Explain why a low best-of-K displacement error is insufficient, and design an evaluation that separates detection, trajectory coverage, probability quality and the resulting vehicle response.
Updated
Example answer
First separate animal detection and tracking from future-motion prediction. Estimate position and motion with appropriate geometry, timing and uncertainty, then consider several plausible futures rather than assuming one path is certain. A neural temporal model is a candidate method, not evidence of reliable collision probability.
Best-of-K displacement error scores only the closest proposed trajectory to the realized one. Adding many implausible alternatives can improve that score without improving probability quality or decisions. Compare coverage and diversity under a controlled prediction budget, relevant missed modes, temporal error, and probability calibration or suitable scoring rules when meaningful probabilities are supplied.
Use independent drives and locations, realistic prevalence and difficult species/occlusion/lighting conditions. Labels and motion reconstruction have uncertainty too. Augmentations must preserve physical geometry, and compression must be retested on relevant cases rather than assumed to keep quality.
Evaluate how the planner uses the predictions, including false hazards and aggressive reactions. A path close to the eventual motion does not show that a proposed response was feasible or safer than alternatives. Authorized closed-loop tests and qualified system review are necessary; the prediction metric alone cannot certify road behavior.