Why LLMs Hallucinate (and What Actually Reduces It)
A technical, no-hand-waving explanation of why large language models make things up: how next-token prediction works, why confidence is not correctness, and the techniques that genuinely reduce hallucination in production.
You have almost certainly seen it happen. You ask a model for a source, and it hands you a perfectly formatted citation: real-sounding authors, a plausible journal, a clean DOI. You go to look it up. None of it exists. The model did not lie to you, because lying requires knowing the truth and choosing to hide it. It did something stranger. It produced the most likely-looking citation, and a fake one fit the pattern just as well as a real one would have.
That is the thing to understand before any fix makes sense. Hallucination is not a bug that slipped into an otherwise truthful system. It is a direct consequence of how the system works. Once you see why, the techniques that actually reduce it stop feeling like magic incantations and start feeling like engineering.
“A language model is trained to be plausible, not to be correct. Those are different targets, and most of the time they happen to agree.
”
What the model is actually doing
A large language model has one core operation: given the text so far, predict the next token, which is roughly a word-piece. It produces a probability distribution over its entire vocabulary, picks from it, appends the result, and repeats. That is the whole loop. Everything impressive it does emerges from doing that very well, billions of times, over patterns learned from an enormous amount of text.
Notice what is missing from that loop. There is no step where the model looks up a fact in a database, checks a source, or asks itself whether what it is about to say is true. It has no ground-truth oracle at inference time. It has a distribution shaped by training, and it samples from it. When the training data strongly supports one continuation, that continuation gets most of the probability and the model is usually right. When the data is thin, conflicting, or absent, the probability spreads out, and the model still has to pick something. It picks the most plausible-looking option, and plausible is not the same as true.
Where hallucinations actually come from
“It makes things up” is not one failure, it is several, and they have different fixes. It helps to name them.
- Knowledge gaps. The fact was never in the training data, or appeared too rarely to be learned. The model cannot retrieve what it never encoded, so it interpolates a plausible answer from nearby patterns. This is the classic fake-citation case.
- Stale knowledge. The model was trained up to a cutoff date and has no awareness of anything after it. Ask about a recent event and it will either decline or confidently describe a world that no longer exists.
- Conflation and interpolation. The model blends two real things into a third that never existed: a real author with a real-sounding but wrong paper, two libraries’ APIs merged into a method that does not exist. The pieces are real, the combination is invented.
- Prompt-induced error. A leading question pulls the model toward agreement. Ask “why is X true” about something false, and a system trained to be helpful will often oblige and manufacture reasons.
- Reasoning slips. Even with the right facts in context, multi-step logic and arithmetic can go wrong, because the model is pattern-matching its way through the steps rather than executing them.
“You cannot fix hallucination in general. You can fix specific kinds of it, and that is a far more tractable problem.
”
What actually reduces it
Here is the honest part that most listicles skip. Nothing eliminates hallucination completely, because it is intrinsic to the mechanism. But several techniques reduce it substantially, and they map cleanly onto the failure types above.
Ground the model in retrieved sources (RAG)
Instead of asking the model to answer from memory, fetch relevant documents and put them in the context, then ask it to answer from those. This directly attacks knowledge-gap and staleness hallucination, because the facts are now in front of it rather than reconstructed from training. It is not a cure: the model can still misread a source or pad the answer with unsupported claims, so pair it with the next step.
Demand citations, then verify them
Require the model to quote or cite the specific passage that supports each claim. This does two things. It biases generation toward statements the sources actually back, and it gives you something checkable. The real win comes when you verify the citations programmatically and reject answers whose sources do not exist or do not say what was claimed.
Lower the temperature for factual work
Sampling temperature controls how much randomness you inject when picking the next token. High temperature is good for brainstorming and bad for facts, because it deliberately chooses less-likely tokens. For factual or extractive tasks, turn it down so the model sticks to its highest-probability, best-supported continuations.
Give it an honourable exit
Models hallucinate partly because they are trained to be helpful and rarely rewarded for saying “I do not know.” Tell it explicitly that declining is a correct answer when the context does not contain the information, and that inventing details is a failure. It will refuse far more often, which is exactly what you want.
Let it use tools instead of guessing
For anything with a real source of truth, do not make the model recall it. Give it a tool: a calculator for arithmetic, a database query for your data, a search call for current events. A model that can call a function stops pattern-matching the answer and starts fetching it, which removes a whole class of reasoning and recall errors.
Add a verification pass
Run the output through a second check: a separate model call that fact-checks each claim against the sources, or self-consistency, where you sample several answers and keep only what they agree on. Disagreement between passes is a strong signal that the model is on thin ice.
A concrete prompt does a surprising amount of the work here, because it reshapes what the model treats as success.
A grounding-and-refusal instruction for a RAG answer step
Answer the question using ONLY the provided sources below.
Rules:
- If the sources do not contain the answer, say exactly: “The provided sources do not cover this.” Do not use outside knowledge to fill the gap.
- After each factual claim, cite the source it came from in brackets.
- Do not infer, extrapolate, or combine sources into claims none of them make on their own.
- If two sources conflict, say so rather than picking one silently.
Sources: […retrieved passages…]
Question: […user question…]
What does not reliably work
It is worth being equally clear about the non-fixes, because teams waste real time on them.
Telling the model “do not hallucinate” or “only say true things” barely helps on its own. It has no internal truth signal to consult, so the instruction has nothing concrete to act on unless you also give it sources and an exit. A bigger or newer model reduces the rate, sometimes dramatically, but it does not change the mechanism, so you cannot treat scale as a guarantee. And a more confident, more fluent answer is not a more correct one. If anything, polish makes hallucinations more dangerous, because it strips away the only surface cue a human might have used to get suspicious.
The fake citation from the opening was never a sign that the model was broken. It was the model doing exactly what it always does, predicting a plausible next token, in a spot where plausible and true had quietly parted ways. Once you accept that, your job gets clearer. You stop hoping for a model that never makes things up, and you start building a system that catches it when it does.
