Can you explain the concept of model drift and its impact on production systems?

Instruction: Define data drift as change in P(X), concept drift as change in P(Y|X), and performance decay as worse measured task loss. Give an example of each and explain why an input-drift alarm alone does not establish model degradation.

Context: This question assesses the candidate's understanding of model drift, which is a critical aspect of model monitoring in MLOps. It evaluates their knowledge of the types of model drift, such as concept drift and data drift, along with their ability to recognize the implications of unchecked model drift on the performance of production models. Additionally, the question probes the candidate's approach to monitoring, identifying, and addressing model drift to ensure sustained model accuracy and relevance.

Updated

Example answer

I would state the terminology used by the team. Here data drift means a change in the input distribution P(X); concept drift means a change in the conditional relationship P(Y|X); performance decay means observed task loss gets worse. 'Model drift' is often used loosely as an umbrella term, so I would avoid treating it as one precise diagnosis.

For example, more requests may arrive from an already well-supported device type, changing P(X) while measured error stays the same. A pricing-policy change can alter the outcome associated with the same features, changing P(Y|X). Loss can also rise because labels or preprocessing are broken, so worsening performance does not by itself identify concept drift.

Track input/schema quality, prediction behavior and mature-label performance by cohort and time window. Investigate meaningful changes against seasonality, traffic mix and label delays. Use drift alarms to trigger diagnosis; do not retrain automatically on an unverified feed. Compare a proposed model or threshold on relevant held-out data, then roll out with monitored rollback criteria.

Related Questions