Synonyms: model confabulation, fabricated content, AI bluffing, plausible nonsense
Do not index
Definition
Hallucination happens when an AI system generates information that sounds believable but is actually incorrect, misleading, or completely made up.
This occurs because large language models predict likely patterns in language, they do not truly “know” facts or verify information automatically.
Use cases
Chatbot misinformation: An AI-powered customer support bot hallucinates a return policy that doesn't exist, leading to user frustration and support tickets when human agents cannot honor the fake policy.
Fabricated research: A product manager uses an AI summarization tool on user interview transcripts, and the tool hallucinates a key pain point that was never mentioned, misguiding the product roadmap.
How it's used in practice
Design for verification: Implement source-citation UI elements. If an AI generates a summary, show exactly which part of the source document the information came from.
Establish confidence levels: When displaying AI outputs, use visual indicators (like probability scores or color coding) to signal how confident the model is in its response.
Implement human-in-the-loop workflows: For critical tasks (like generating medical advice or legal content), require AI-generated drafts to be reviewed and approved by a human expert before final delivery.
🪄
Pro-tip: ever assume users understand how LLMs work. If your product includes generative AI, user education becomes part of the UX. Explicitly teach users about the possibility of hallucinations and how to spot them.
Challenges & limitations
Identifying hallucinations: It is often difficult for users (and sometimes developers) to distinguish between a correct AI response and a plausible-sounding hallucination without manual fact-checking.
Model unpredictability: Even with strict prompting, models can still hallucinate, making it difficult to create a consistently reliable user experience.
