How can you Prevent ML Models from Degrading?

By CIOReview | Thursday, August 1, 2019

ML models tend to degrade with time, which can be prevented by monitoring the performance of the models after deployment.

FREMONT, CA: Machine learning (ML) models are transforming the business processes across the industries. From manufacturing to supply chains, each vertical is leveraging ML in some ways. However, the important question is, how long are the models self-sustainable? Assuming that the ML solution will work forever on its own is illogical. The reality is ML models tend to degrade with time.

Why ML Models Degrade with Time?

Starting with relevant data that enables accurate predictions is fine, but the degradation starts from the time the system is deployed with the data. The phenomenon is called as concept drift, and despite being studied in academia for the last two decades, it has still not been given get its due importance in the industry.

     Check out: Top Machine Learning Solution Companies

Concept Drift

According to Concept drift, the statistical properties of the particular variable that the model is trying to predict changes over time. It happens because the predictions become less accurate over time. In contrast to a calculator, an ML system interacts with the real world. As per concept drift, the interpretation of the data changes with time despite the data distribution, in general, being the same. Thus the end-user tends to interpret the model predictions as having deteriorated over time.

Preventing Model Degradation

It is crucial to monitor the performance of the ML model after deployment. If monitoring all features requires too much effort, the deployment team must identify the key features whose variation in data distribution can affect the model results considerably.

Moreover, a degraded model does not necessarily by restructured all over again. Instead, creating a retrained model by incorporating additional features would be a better alternative that can enhance the model’s performance and make it more sustainable for the future.

     Machine Learning Solution Companies - Elliptic LabsFaethmMegvii

After the above steps are taken, it is time to recreate the model using the new or modified features and model parameters. The focus here is to work towards a model that delivers the best accuracy and generalized well to a few data drifts.

In some cases, a re-creation of models doesn’t improve model performance. Analyzing the examples that the model got wrong and searching for trends that are beyond the current feature set can aid in identifying new features.