Why can a model score well on validation and fail after deployment?
Instruction: A churn model uses refund_processed_at, a field populated after the prediction cutoff. Random row splitting also places multiple records for the same customer in both train and validation. Validation AUC is 0.99; a later-month deployment sample performs poorly. Propose a corrected evaluation.
Context: Diagnose target and entity leakage before applying model regularization.
Updated
Official answer available
Read the opening below, then unlock the full answer and practical guidance.
There are two concrete leakage paths. Remove refund_processed_at and any derived feature unavailable at the prediction cutoff; reconstruct every feature as it would have been known then...
Related Questions
-
easy
-
easy
-
easy
-
medium