Trust calibration is the practice of designing AI interfaces so users trust the system about as much as it deserves. The aim is to close the gap between how reliable an output actually is and how reliable it feels, so people don't over-rely on wrong answers or ignore correct ones.
Use cases
When trust is miscalibrated, people make bad calls fast and quietly.
Overtrust in copilots: A user accepts a confident-sounding legal or medical answer without checking it. The tone read as authority, so nobody verified the claim. One wrong output that feels certain causes more harm than a vague one.
Useless confidence scores: A "92% confident" label sits next to a wrong answer. After two or three of these, users learn to ignore the number entirely, and the signal is dead.
How it's used in practice
Tie confidence display to measured accuracy: Only show "high confidence" where tested outputs are actually right that often. Guessed numbers train users to distrust all of them.
Add verification affordances: Show sources, citations, or a quick "check this" path so users can confirm before they act on something.
Scale friction to stakes: Let low-risk actions apply automatically. Require a confirm step for high-risk ones like sending money or deleting data.
Test with real failures: Prototype with live model outputs, including the bad ones, instead of clean mockups that hide where the model breaks.
🪄
Pro-tip: A confidence number on a wrong answer does more damage than no number.
If you can't prove that "90%" on screen means 90% correct in production, drop the number and use qualitative cues instead, like visible sources and a prompt to double-check.
Challenges & limitations
You need ground truth: Honest confidence requires accurate data on real cases, which most teams don't have early on. Without it, any confidence signal is a guess.
Calibration drifts: A signal that's accurate today breaks after a retrain or a silent switch to a cheaper model. It needs ongoing testing, not a one-time setup.
