Guessing Was The Winning Strategy

The case that did not exist

A lawyer files a brief citing several cases. The cases have names, courts, dates, and reporter numbers. They are formatted perfectly.

Several do not exist.

This has now happened enough times to be catalogued. Public trackers of court decisions involving fabricated AI citations record well over a thousand incidents, with sanctions ranging into tens of thousands of dollars.

The question worth answering is not how someone could be so careless. It is why the model produced perfectly formatted references to nothing.

Not lying, and not confused

Two explanations are tempting and both are wrong.

It is not lying. Lying requires knowing the truth and choosing otherwise. There is no separate store of truth being consulted.

It is not confused. Confusion implies an expectation of accuracy that failed. The system is doing exactly what it does: producing likely text.

A legal citation has an extremely regular form. Case name, versus, case name, volume, reporter, page, court, year. That pattern is enormously well-represented in training text.

Generating a plausible instance of that pattern is easy. Whether the instance corresponds to a real case is a question the mechanism never asks.

Why it does not say it is unsure

This is the part with a satisfying explanation.

Models are evaluated on benchmarks, and benchmarks score answers. Consider two models with identical knowledge:

Model A answers when confident, abstains when not. Model B always answers, guessing when unsure.

On the questions both know, they tie. On the questions neither knows, A scores zero and B scores whatever its guesses happen to earn, which is more than zero.

B looks better. B gets selected for. So the training and evaluation regime systematically rewards guessing over honesty, and models have been shaped by it.

This is now the leading account of why hallucination persists, and it reframes the problem: fabrication is not a defect in the technology so much as an artifact of how we score it.

The consequence

If it is an incentive problem rather than a capability gap, then waiting for it to be fixed is a poor plan.

It can improve. Better grounding helps, tool use helps, and evaluations that reward calibrated uncertainty would help most of all.

But as long as scoring rewards answering, models will lean toward answering. Design your habits for a system that guesses, because that is the system you have.