Back to Glossary
Model Drift
Synonyms: model decay, model degradation, AI drift, LLM drift, performance drift
Do not index
Definition
Model drift is when an AI model's answers get worse over time, even though no one touched your product. Real-world inputs move away from the training data (data drift), or the right answer changes (concept drift). With LLMs, the provider can also update the model under your prompt.
Use cases
Drift doesn't crash anything, so teams find out from churn and support tickets.
- The prompt that stopped working: In a Stanford and UC Berkeley study, GPT-4 scored 84% on a prime-number test in March 2023 and 51% in June. Products using that prompt got worse with zero code changes.
- The "better" model that broke trust: Microsoft researchers describe a diagnosis model that went from 95% to 98% accuracy and started missing older patients, a group the old model got right.
How it's used in practice
- Pin the model version: Call a dated model snapshot and upgrade on purpose, after testing.
- Keep a golden test set: Save 50 to 100 real user inputs with approved answers. Rerun them after every model or prompt change.
- Track inputs and outputs in production: Compare live inputs and predictions against your launch baseline.
- Give users a way to flag bad answers: A thumbs-down button turns complaints into an early drift alert.
Before you switch models, list the inputs the old model got right. Overall accuracy can rise while those cases break, and users already trust those cases. Check them first.
Challenges & limitations
- Late answers: A loan model's mistake shows up months later, when the loan defaults. Until then, you're guessing.
- False alarms: Inputs can shift with no effect on quality, and teams learn to ignore the alerts.
- Vendor schedules: OpenAI and Anthropic retire models on their own dates, and you migrate anyway.
Free resources
- What Is Data Drift in ML, and How to Detect and Handle It (Evidently AI) — input shifts, how they differ from concept drift, and how to respond
- Data Distribution Shifts and Monitoring (Chip Huyen) — Stanford CS 329S course notes on why deployed models degrade
