Every production machine learning system is susceptible to covariate shift, when the distribution of production data predicted on at run time “drifts” from the distribution you trained on. This phenomenon can severely degrade a model’s performance and can occur for a variety of reasons - for example, your company now sells new products or no longer offers old ones, the price of inventory has increased or there is a bug in the software that sends data to your model. The effect of covariate drift can be subtle and hard to detect but nevertheless important to catch. In Google’s machine learning guide they report that refreshing a single stale table resulted in a 2% increase in install rates for Google Play. Retraining and redeploying your model is an obvious fix, but it can be ambiguous as to how often you should do this, as soon as new data arrives, every day, every week, every month? What’s more this would only work for the first two examples mentioned above and can’t be used to fix integration bugs.