Why AI Models Hallucinate

Hallucination isn't a malfunction: benchmarks reward confident guessing and penalise "I don't know", so training selects for it. Purpose-built retrieval tools sold as hallucination-free still fabricate 17–33% of the time.

Reviewed

A hallucination is not the model malfunctioning. It is the model doing exactly what it was trained and evaluated to do, in a case where that produces a false answer. Understanding why is the difference between a workflow that manages the problem and one that is quietly broken.

The training explanation

In Why Language Models Hallucinate (September 2025), Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang make an argument that is uncomfortable because it is structural rather than technical.

Benchmarks overwhelmingly score answers as right or wrong. I don't know scores zero. A guess might be right. So across the enormous number of evaluations that shape a model's post-training, the process systematically rewards guessing over expressing uncertainty.

Their analogy is a student sitting a hard exam. Leaving it blank guarantees no marks; a plausible guess might land. Any student optimising for the score guesses. We built the same incentive into model evaluation, and then we are surprised that models are confident when they should not be.

The important implication: this is not a defect that better pre-training removes. As long as the scoring rewards confident answers and penalises abstention, the behaviour is being selected for. Fixing it means changing how models are graded, not just how they are built.

How bad it actually is

The best-measured domain is law, because legal claims are precisely checkable — a case either exists or it does not.

In Large Legal Fictions, published in the Journal of Legal Analysis in 2024, Matthew Dahl, Varun Magesh, Mirac Suzgun and Daniel E. Ho found hallucination rates of 58% to 88% across leading models on specific legal queries. Two secondary findings matter more than the headline number:

The models did not know when they were wrong. There was no reliable internal signal separating a correct answer from a fabricated one.

They reinforced false premises. Asked a question built on an incorrect assumption, models tended to accept the assumption and build on it — which is precisely the failure mode of someone researching a topic they do not already understand.

Retrieval helps. It does not solve it.

The obvious response is to ground the model in real documents. And it works — but the size of the remaining gap is the single most useful number in this whole subject.

Stanford's RegLab tested the purpose-built, retrieval-augmented legal research tools sold by LexisNexis and Thomson Reuters — commercial products, marketed as having eliminated this problem. They found the tools hallucinate between 17% and 33% of the time.

Sit with that. These are not general chatbots. They are systems built specifically for this task, by companies with the best legal corpora in existence, sold on the explicit promise of being hallucination-free. Retrieval cut the error rate substantially and left something between one-in-six and one-in-three.

If you are designing a workflow on the assumption that RAG makes output trustworthy, that assumption is measurably wrong. RAG vs fine-tuning vs prompt engineering covers what retrieval is genuinely for; making output verifiable is not the same as making it correct.

Why you cannot spot them

The property that makes hallucinations dangerous is that they are produced by the same process as correct answers, so they carry the same surface markers of quality.

A fabricated citation has a plausible author, a plausible journal, a plausible year, and correct formatting. A fabricated statistic has a plausible magnitude and a named source. Fluency and confidence are generated identically whether the underlying claim is true or false — which means neither carries any information about correctness.

And the exposure is worst exactly where you are least equipped to notice, because you are asking about something you do not know. Everyone's instinct is to trust an answer that reads well. That instinct is calibrated for human writers, where fluency correlates with expertise. It does not transfer.

What actually reduces it

Supply the source material. The single biggest reduction available. A model summarising a document you provided is doing a fundamentally more reliable task than a model recalling a fact. Most of the research prompts are built on this rule: work from material you supply, never generate the facts.

Ask for citations you can check — then check them. Citations do not make output true; they make it falsifiable, which is different and more valuable. An uncheckable claim cannot be caught. Follow every one that matters.

Make I don't know a legitimate answer. Explicitly instruct the model to say when it lacks the information, and — this is the part people skip — do not treat that as a failed response when it happens. If your prompt or your product penalises abstention, you have recreated the benchmark problem locally.

Separate the claim from the inference. Prompts that force output into what the source says versus what I concluded make the fabricated parts visible, because inventions cluster in the second category.

Never ask for facts you could look up. Prices, dates, headcounts, competitor details, current events. The model has no access and will produce something shaped like an answer.

Constrain the output. A schema-validated response has fewer degrees of freedom to invent in — see getting reliable structured output.

Measure it on your own data. General benchmarks tell you little about your use case. How to write evals for your AI agent and how to evaluate and test your prompts cover building a set with known-correct answers, which is the only way to know your actual rate.

What does not work

Asking the model if it's sure. Self-assessment fails for the reason the legal study identified — there is no reliable internal error signal to report.

Turning the temperature down. Lower temperature makes output more deterministic, not more true. It will produce the same fabrication more consistently.

Assuming the next model fixes it. Rates have improved. The mechanism has not changed, and the Stanford result shows that even purpose-built systems with excellent retrieval retain a substantial rate.

The working assumption

Treat every factual claim, figure, name, quotation, and citation as unverified until you have checked it against a primary source — and treat the confident, well-formatted ones with more suspicion rather than less, because that is exactly what a fabricated one looks like.

That is not a counsel of despair. Structuring an enquiry, organising evidence, surfacing what you have missed, and stress-testing your reasoning are all things these tools do genuinely well, and none of them require the model to be an authority on the facts. The research and analysis prompts are built entirely around that division of labour: the model does the structure, you do the verification.

Sources

← Back to Articles How we research & review →