After deploying an ML model, you monitor its performance and notice that the model's accuracy has steadily dropped below your acceptable threshold. Surprisingly, your statistical checks show that the distribution of incoming inference data has remained identical to the training data. What MLOps phenomenon are you observing?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: you build a model, it runs perfectly in the lab, and you deploy it to production. But over time, its accuracy starts slipping. You check the incoming data, thinking 'Aha! The users are sending us different data now!' But nope, the data looks exactly the same as it did on day one. So what gives? This is called model degradation. In this case, it's often caused by what we call 'concept drift'—the relationship between the input features and the target labels has changed. For example, if you're predicting home prices, and a new interest rate hike suddenly changes how much people are willing to pay for a house, your inputs (square footage, bedrooms) haven't changed, but the real-world outcome has! Your model is now out of date and needs a serious retraining session. Don't let it sit there and rot!
Full explanation below image
Full Explanation
Model degradation (or model decay) refers to the decline in a machine learning model's predictive performance after it has been deployed to production. Even if the incoming feature distribution remains constant (meaning there is no data drift or covariate shift), a model can still degrade due to concept drift. Concept drift occurs when the statistical properties of the target variable change over time relative to the input variables. In other words, the mathematical relationship between the inputs ($X$) and the outputs ($y$) has shifted.
Common causes of model degradation include: 1. Concept Drift: Changes in user behavior, macroeconomic conditions, or seasonal trends that alter the real-world outcomes associated with specific inputs. 2. Software/Environmental Degradation: Changes in dependencies, API updates, or hardware differences that cause the model's software environment to behave differently. 3. Feedback Loops: When the model's own predictions influence future inputs, creating a biased data loop that degrades performance over time.
To combat model degradation, MLOps engineers set up continuous monitoring systems that track model performance metrics (such as accuracy, F1-score, or MAE) against ground truth labels as they become available. Once a model's performance falls below a predefined threshold, it triggers automated alerts or retraining pipelines to update the model using the most recent data.