Ask for a source and you might get a beautifully formatted citation to a paper that does not exist. The model was not lying. It was doing exactly what it always does: producing the most plausible next words.
When the model knows the answer cold, plausible and true line up. When it does not, plausible fills the gap anyway, in the same confident voice.
Why it happens
Language models are rewarded for producing likely text, and the likely shape of an answer includes specifics: names, dates, numbers, sources. When the true specifics are missing from what it learned or what it was given, the shape gets filled with invented ones. Nothing inside flags the difference.
The gap can come from two places. Sometimes the model never learned the fact. Sometimes nobody gave it the document. Both produce the same fluent guess, which is why pasting the source changes everything: the model stops trying to remember and starts reading, and reading is far more reliable.
When it happens most
Narrow questions about obscure things. Exact quotes, prices, and citations. Anything that changed recently, since the model’s training has a cutoff. Long chains of reasoning where one early slip compounds. Recognizing these zones is most of the defense.
There is a strange flip side to this. Made-up answers are rarest on common knowledge, where every source agrees and you barely needed AI in the first place. They are most common on obscure specifics, which is exactly where you are most tempted to trust the machine because you cannot easily check it yourself. When a question is obscure, treat that as the warning light.
What actually helps
Grounding. Give the model the document instead of asking it to recall, or use tools that search and cite so claims trace to sources. Then check the sentence you are about to act on: the price, the dose, the deadline. One targeted check beats rereading the whole answer.
The cheapest defense is to ask for sources up front, and to get suspicious when an important claim shows up without one. A system that actually found the answer can show you where. A system that guessed will make up a source too, and a fake citation takes about one search to expose.