In 2023, a lawyer in New York asked ChatGPT to help research a case. It came back with six court decisions that supported his argument, complete with names, dates, and quotes. He used them in an official court filing. There was just one problem: none of those cases existed. ChatGPT had made every single one up, and when the lawyer asked it to double check, it confidently assured him they were real.
The judge was not amused. Both attorneys were fined, and the story became one of the most famous cautionary tales in AI. But here's the part that matters most for the rest of us: the lawyer wasn't careless or unusually gullible. He ran into something every single AI chatbot does, called a hallucination, and most people have no idea it's even possible until it happens to them.
So what is a hallucination, exactly?
An AI hallucination is when a chatbot states something false as if it were fact, with zero hesitation and often with impressive-looking detail attached. It's not lying, because lying requires knowing the truth and choosing to hide it. AI doesn't know what's true. It's something stranger: a system that is very good at sounding right, generating an answer that fits the shape of a good answer, whether or not the content behind it is real.
That's what makes hallucinations so easy to miss. A wrong answer that sounds unsure is easy to catch. A wrong answer delivered with total confidence, in full sentences, with a citation attached, is not.
Why AI does this in the first place
Tools like ChatGPT, Claude, and Gemini don't have a database of facts they check against. Underneath, they're predicting the next most likely word based on patterns learned from enormous amounts of text. Most of the time that produces something accurate, because accurate information tends to be common in what the model learned from. But when a model doesn't actually know something, the underlying mechanism doesn't stop and say so. It keeps predicting the next likely word anyway, and the result reads exactly as smoothly as a true answer would.
It gets worse for very specific requests. A general question about a topic pulls from a wide, well-covered pattern. A request for a specific court case, a specific statistic, or a specific citation asks the model to be exact about something narrow, and narrow is exactly where these systems are most likely to fill in a gap with something plausible-sounding instead of admitting they don't know.
There's a training reason this gets worse, not better, without deliberate effort. Many AI models are fine-tuned using human feedback, where people rate its answers. A confident, complete-sounding answer usually rates better than one that says "I'm not sure." Over time, that nudges the model toward sounding certain, even in moments when certainty isn't earned.
The chatbot isn't trying to deceive you. It's built to sound helpful, and sounding helpful and being correct aren't automatically the same thing.
It's not just lawyers
The court case above is the most famous example, but it's far from the only one. In February 2024, an airline customer named Jake Moffatt asked Air Canada's website chatbot about bereavement fares after his grandmother passed away. The chatbot told him he could book a full-price ticket and apply for a retroactive discount within 90 days. He did exactly that, and Air Canada refused the refund, because that policy did not exist. The chatbot had invented it.
Air Canada tried arguing in the tribunal that the chatbot was responsible for its own words, not the company. The tribunal disagreed and ordered Air Canada to pay damages, establishing a precedent that still gets cited today: if your AI tells a customer something false, the company is on the hook for it, not the bot.
Both stories share the same root cause. A tool that sounds authoritative got trusted the way you'd trust a search engine or a reference book, when what it actually needed was a second check.
How to spot one before it costs you
You don't need to distrust every answer an AI gives you, but a few habits go a long way, especially for anything with real consequences:
- Treat specifics as unverified until you check them. Names, dates, statistics, quotes, and citations are exactly where hallucinations like to hide.
- Ask "how do you know that?" A model that can point to a real, checkable source is on firmer ground than one that just restates its answer more confidently.
- Be extra careful with anything narrow or recent. The more specific or obscure the request, the more room there is for the model to guess.
- Search it yourself for anything that matters. If a decision involves money, health, legal standing, or a grade, a two-minute independent check is worth it every time.
- Notice confidence isn't evidence. A calm, detailed tone feels trustworthy, but tone is not the same thing as accuracy.
Hallucinations aren't a bug that will just quietly disappear as models improve. They come from how these systems fundamentally work, predicting likely text rather than looking facts up. Newer models hallucinate less often, but "less often" still means it will happen to you eventually.
The takeaway
None of this means AI tools aren't useful. Both the lawyer and the airline chatbot could have avoided real damage with one extra step: checking the specific, checkable facts before acting on them. That's really the whole skill. Use AI to draft, brainstorm, and explain, and treat anything specific and consequential as a claim to verify, not a fact to repeat.
That's the kind of thing we mean when we say AI literacy. Not fear of the tools, and not blind trust in them either. Just understanding well enough to use them with your eyes open.