A language model that hallucinates is not malfunctioning. It is doing exactly what it was trained to do. Next-token prediction optimizes for one thing — the statistical plausibility of the next token given everything before it — and "is this true" is not a term in that objective. When the true continuation is uncertain but a fluent, confident continuation is available, the objective rewards the fluent one. A well-formed falsehood delivered with conviction is not a failure of the loss. It is the loss doing its job.
This matters because the dominant framing treats hallucination as a bug: something to be caught, patched, and eventually eliminated by a bigger model or a cleverer prompt. That framing leads to the wrong engineering. If you believe the defect lives in the model, you wait for a fix. If you understand that it lives in the objective, you stop waiting and start designing around a confident, fallible reasoner — a problem, as I'll argue, that other disciplines already know how to manage.
The objective has no term for truth
Pretraining minimizes cross-entropy. Concretely: for a context c and a next token t, the model is trained to minimize the expected value of −log p(t | c) over the corpus. Gradient descent pushes the model's conditional distribution p(· | c) toward the empirical conditional distribution of the training text. That is the entire signal. There is no oracle for the world anywhere in that expression.
Read the consequence carefully, because it is the whole essay. The only ground truth the gradient ever sees is which token actually came next in this document. The model is therefore not learning what is true. It is learning what is likely to be written. Most of the time those two point the same way — accurate text is more common than any single specific falsehood, so predicting the corpus predicts a lot of true statements for free. That coincidence is why the models are useful at all. But it is a coincidence of the data distribution, not a property the objective enforces. Wherever the corpus is thin, contradictory, or simply silent about the specific thing you asked, the coincidence breaks and the objective keeps optimizing the only thing it ever optimized: plausibility.
Take the canonical case, a fabricated citation. The model has seen the pattern [Author] et al. ([Year]) tens of thousands of times. Ask it for a reference and, token by token, it emits the most probable surname given the topic, the most probable co-authors given that surname, the most probable year given that cluster of names, the most probable journal given the field. Each token is individually the high-probability choice. The result is a syntactically flawless citation to a paper that does not exist. Nothing went wrong. Every token minimized loss. The model never had access to a channel that could tell it whether the paper is real, because "real" is a relation between a string and the world, and the objective only ever compared strings to other strings.
Sampling turns a distribution into a claim
There is a second mechanism stacked on top of the first, and it is easy to miss. Even if the model's internal distribution honestly reflected its uncertainty, generation destroys that information.
At the moment the true continuation is uncertain, the model might hold a genuinely diffuse distribution — say 20% on one answer, 15% on another, a long tail beyond. That diffuseness is the model representing "I'm not sure." But decoding collapses the distribution into a single surface string. Greedy decoding picks the mode; sampling draws one option. Either way, the uncertainty that lived in the probabilities does not survive into the text. The output "The melting point is 1,412°C" carries the same grammatical confidence whether the model assigned that answer 0.95 or 0.09. The reader receives a claim; the model emitted a sample. The interface between them silently launders a probability into an assertion.
This is also why a text-only model cannot rescue itself by "thinking harder." To check a generated string against reality, you need a channel to reality. The model has a channel to its own compressed statistics of the corpus, which is a different thing — it can tell you what is typically said, not what is the case. That gap is the subject of a separate argument I've made about why fluent continuation is not comprehension; see Prediction Is Not Understanding. Here the narrower point suffices: a system whose only feedback signal is corpus likelihood has no internal operation that corresponds to verification.
RLHF trades calibration for confidence
Now the twist that makes deployed models worse than their base models on exactly this axis.
Reinforcement learning from human feedback fine-tunes the model to maximize a reward model trained on human preference comparisons. This is what makes assistants helpful, on-topic, and pleasant. It also degrades calibration, and the effect is documented rather than speculative: OpenAI's GPT-4 technical report shows the pretrained model was well calibrated on multiple-choice questions — its stated confidence tracked its actual accuracy — and that this calibration got noticeably worse after post-training with RLHF.
The mechanism is not mysterious. Human raters, and the reward models that imitate them, prefer confident, direct, agreeable answers. In a pairwise comparison, a hedged "I'm not certain, but possibly X" reliably loses to a crisp "It's X" — including in the many cases where the rater cannot themselves tell that X is wrong. So the gradient of the reward teaches a specific lesson: expressed uncertainty is penalized relative to a plausible guess. The model learns to suppress the diffuse-distribution signal that was its one honest cue and to speak in the register that wins comparisons. You optimize for helpfulness and you get bluffing as a side effect, because at the point of uncertainty the two are behaviorally identical.
A 2025 OpenAI paper, "Why Language Models Hallucinate" (Kalai, Nachum, Vempala, and Zhang), pushes this to the evaluation layer, and it is the sharpest version of the argument, so let me state it in its strongest form. Benchmarks are graded like school exams: a right answer scores one, a wrong answer scores zero, and an abstention — "I don't know" — also scores zero. Run the arithmetic from the model's side. If its best guess has any probability p > 0 of being marked correct, then the expected score of guessing is p, and the expected score of abstaining is 0. Since p > 0, guessing dominates abstention under that scoring rule. It is never rewarded to say "I don't know" on such a test. We have therefore been selecting models, across both training and public leaderboards, for their willingness to bet on a plausible answer rather than admit ignorance. Hallucination is not smuggled in against our incentives. It is the equilibrium those incentives produce.
The mitigations are real, and none of them touch the root
Serious work reduces hallucination in practice. It just doesn't remove the cause, and knowing the difference tells you where to spend effort.
| Mitigation | What it adds | Why the root survives |
|---|---|---|
| Retrieval / grounding (RAG, Lewis et al. 2020) | An external document signal the base objective lacks | The model can still misread, over-extrapolate, or answer fluently when the documents are silent |
| Abstention training | An "I don't know" option and reward for using it | Moves the operating point but reintroduces the calibration problem; only works if evaluation rewards abstention too |
| Calibration methods (e.g. temperature scaling, Guo et al. 2017) | Better-behaved reported probabilities | Fixes the number, not the generated claim; decoding still collapses it to a confident string |
| Verification / semantic entropy (Farquhar et al., Nature 2024) | A detector that flags likely confabulation via meaning-level disagreement across samples | It is a detector, not a cure; the generator still optimizes plausibility |
Look at what the whole column shares. Every one of these methods either bolts an external signal onto the system or runs a second process to audit the first. Not one alters the generative objective, because the generative objective is not a thing you patch — it is the definition of the model. You are building scaffolding around a confident, fallible reasoner. That is worth doing. Just don't confuse it with repairing the reasoner.
A solved management problem, even where the modeling problem is open
"Confident, fallible reasoner producing plausible answers under uncertainty" is not a novel object. It is precisely what clinical medicine has spent a century learning to operate around, and the transfer is nearly one-to-one. A physician's differential diagnosis is a discipline built on the assumption that the first plausible answer is dangerous: you enumerate competing hypotheses, weight them by base rate, and — this is the load-bearing move — order tests chosen to disconfirm, not confirm, the leading candidate. The goal is never to make the clinician infallible. It is to build a process that catches the confident wrong answer before it acts on a patient. I've drawn that parallel out in full in what a physician's differential diagnosis teaches about LLM hallucination; the lesson that carries here is that you do not fix an unreliable estimator by wishing it reliable. You wrap it in a protocol that expects it to be wrong.
Three commitments follow, and they are the opposite of waiting for a better model.
Route factual claims to external verification. If a claim can be checked against a database, a retrieval index, a calculator, or a second independent process, check it — and treat the model's own assent to its own output as worth nothing, because it is the same objective grading itself.
Reward abstention where it actually counts: in your evaluations, not just your system prompt. Adopt scoring that penalizes a confident wrong answer more than an honest "I don't know." The moment your metric stops paying out for lucky guesses, you start selecting for models that abstain, and the incentive that manufactures hallucination weakens at its source.
Never read confidence as truth. Fluency is the exact quantity the objective was built to maximize, which makes it the least informative signal you have about correctness. A model's certainty and a model's accuracy are produced by different mechanisms, and RLHF actively pried them apart.
There's an old philosophical name for speech produced to fit a situation without regard to whether it's true. Harry Frankfurt called it bullshit, and distinguished it sharply from lying: the liar tracks the truth in order to invert it, while the bullshitter is simply indifferent to it, optimizing for the utterance that best serves some other goal. Read as engineering rather than insult, that is an accurate spec sheet for a next-token predictor. Its goal is the most fitting continuation, where "fitting" means probable-and-preferred, not corresponding-to-the-world.
The model is not trying to deceive you. It was never trying to tell you the truth. That was your category, not its objective.