When AI makes things up
How to spot a confident-sounding wrong answer.
By the end of this lesson
After this lesson, you'll be able to spot the difference between confident and correct, and run a 30-second check on any AI answer that matters.
Most of us have had this happen. You ask an AI for something — a recipe, a fact, a citation, a recommendation. The answer comes back fast, polished, confident. You believe it. Then it turns out to be wrong.
It's not lying. There's no intent. The AI is built to predict the next plausible word, not to know what's true. The wrong answer arrives sounding exactly like a right one. That's the trap.
The plausible answer that isn't true
In 2023, a New York lawyer used ChatGPT to prepare a brief for court. ChatGPT cited six previous cases that perfectly supported his argument — judges, courts, quotations, all detailed. The lawyer filed the brief. The judge looked up the cases. None of them existed. ChatGPT had invented all six.
The lawyer wasn't trying to cheat. He thought ChatGPT was searching a legal database. It wasn't — it was predicting what plausible legal citations look like, given his question. Plausible isn't the same as real.
This is called a hallucination — when an AI says something untrue with full confidence, as if it were a fact. It's not rare. It's a feature of how the technology works.
When your friend confidently recommends a cafe in town that turns out not to exist, they aren't lying. They remember it that vividly. AI is the same — it doesn't know the difference.
Why the tone is the trap
Wrong answers from AI sound exactly like right ones. Same calm sentences. Same neat structure. Same confidence.
That's because the AI doesn't have a separate "I'm not sure" setting. It generates one word at a time, picking the most likely next word given everything before. There's no "is this true?" check inside the loop.
So if you ask "is this answer right or wrong?" from looking at the words alone — you can't tell. The only way to know is to look outside the AI.
If a friend told you something completely wrong but said it with the same tone they use for things they actually know — you'd believe them. AI is exactly that friend.
The 30-second cross-check
For anything that matters — money, medicine, law, dates, a name, a number — there's a habit that catches almost everything wrong. It takes 30 seconds.
- Search the most specific fact. Take a name, date, or number from the answer and Google it. If it's real, you'll find it from named, independent sources. If not, you'll find only AI-shaped websites repeating each other.
- Click any source the AI cited. If the link doesn't load, or the page doesn't actually say what the AI claimed — that's your answer.
- Ask the AI: "What's the strongest reason this might be wrong?" A useful answer will show weak spots. A risky one will just repeat itself with the same confidence.
If your GP told you a drug is safe to mix with another, you'd ask "where's that documented?" before doing it. Same friend, same caution — applies to AI.
Try this for 2 minutes
Open ChatGPT (free at chat.openai.com). Ask:
"Recommend a small independent bookshop in [your suburb or town]. Give me the name, the street, and a recent review."
Read the answer carefully. Now try to verify the bookshop. Search the name. Check if the street exists. Look for the review.
Sometimes you'll find a real shop. Often you won't — the AI invented one that sounds plausible for your area. That's a hallucination, in your hands, in 90 seconds.
Keep this
- Rule — If an AI answer matters — even a little — find a second source for it before you trust it.
- Phrase — "Where did you get that?" — applies to AI exactly like to a friend.
- Don't — Don't quote an AI answer to a colleague, your kid's teacher, or your doctor as if it were a source.
Pop quiz, no marks
-
Your aunt says ChatGPT confirmed a herbal tea is safe with her blood pressure medication. What's your reply?
Show answer
Lovely — but let's check that with the pharmacist or her GP. AI gets medication confidently wrong, and the tone doesn't change when it's wrong.
-
AI gave you three sources for an essay. What do you do before quoting any of them?
Show answer
Click each link. If it doesn't load, or doesn't say what the AI claimed, the source is invented — even if it sounds real.
Want to go deeper?
Each of these topics on Plain AI explores one idea from this lesson in more detail: