How to Use AI Without It Making Things Up

Ask an assistant a question and you get an answer that is fluent, structured, and delivered with total composure. That composure is the problem. Nothing in the tone tells you which parts it knows and which parts it assembled because they sounded right.

The usual advice is to watch out for AI hallucinations, meaning confidently stated things that are not true. That advice is now slightly out of date, because the failure has moved. Models rarely invent whole objects any more. They invent one small detail inside an otherwise correct answer, which is far harder to catch and far more likely to end up in your work.

There is a legal marker for how seriously this is being taken. In May 2026 the Regional Court of Munich ruled that Google is directly liable for false claims in its AI search summaries, rejecting the argument that users should verify the output themselves. In the same coverage, an analysis for the New York Times found Google's summaries were correct about 91% of the time, and that 56% of the correct answers could not be supported by the sources Google had linked underneath them.

Read that second number again. Being right and being sourced turned out to be two different things.

The old test doesn't work any more

I tried the classic demonstration first. I asked ChatGPT about a character in a Kazuo Ishiguro novel called The Winter Almanac, published in 1998. No such book exists.

It did not play along. It searched, told me there is no Ishiguro novel by that title, pointed out that his publication run goes from The Unconsoled in 1995 to When We Were Orphans in 2000, and offered to identify whichever book I was actually thinking of.

That is a real improvement, and it matters, because the demo everyone still repeats is out of date. On an obvious false premise, a current model with reasoning and search switched on will usually push back rather than invent.

The research shows the same shape. Walters and Wilder generated 84 literature reviews and checked all 636 citations in them. 55% of the older model's citations were fabricated, against 18% for the newer one. Fabrication fell by two thirds.

But look at their other number. Among the citations that were real, the rate carrying a substantive error only fell from 43% to 24%. Wholesale invention got much rarer. Wrong details inside real answers barely moved.

[IMAGE: ChatGPT correcting a false premise about a non-existent Kazuo Ishiguro novel, with a Google Books source chip visible under the answer]

So I went looking for the newer failure

I asked the same assistant for three academic papers measuring how often AI fabricates citations, with authors, years and direct links, and told it to answer from memory without searching.

It produced three. Then I checked every one against Crossref, the official registry that publishers deposit their records into.

All three papers were real. All three links resolved. The titles were exact, the years were right, the journals were right. On any normal spot check this answer passes.

One author's first name was wrong. The third paper is by Jocelyn Gravel, Madeleine D'Amours-Gravel and Esli Osmanlliu. I had been given Elie Osmanlliu. A second author was given as "Esther I. Wilder" where the record says Esther Isabelle Wilder.

Nothing was invented. One letter cluster was quietly replaced inside an otherwise flawless citation, and it is exactly the kind of detail nobody re-reads. If you had pasted that into a reference list, it would have survived every check short of opening the paper.

That is the modern failure mode, and it is why "does this look right" stopped being a useful test.

The routine that actually catches it

Five habits. None of them takes long, and they work on any assistant.

1. Say where the answer should come from. "Search for this and answer only from what you find" changes the job from recall to retrieval. It is the single highest-value instruction in this list, because a model answering from memory is reconstructing, and reconstruction is where details drift.
2. Ask for the claim and the source as separate things. Request the answer, then the specific page it came from, then open the page. The Oumi finding above is the whole argument for this step: more than half of Google's correct summaries could not be traced to the sources sitting right under them. A link is not a citation until someone reads it.
3. Make it mark its own uncertainty. Add "flag anything you are not confident about, and say why" to the end of your prompt. In my test the assistant volunteered, unprompted, that it was most confident in the first two citations and that the third was worth checking independently. It was right, and that is where the error was.
4. Run a second pass as a separate turn. This is the strongest move and almost nobody does it.
5. Verify the one detail you will act on. Not everything. The number you will put in the email, the date you will book around, the name you will publish. Checking one fact properly beats skimming 10.

The second-pass prompt, verbatim

After I had the three citations, I sent this as a fresh message:

Now search and check your own three citations against the actual records. For each one, tell me whether the title, every author name, the year and the DOI are exactly right, and flag anything that is wrong. Be specific about which detail is wrong.

It came back with a table, ticks against everything correct, and a red cross against exactly two rows: "Elie" should be "Esli", and "Esther I." should be "Esther Isabelle". Those were the same two errors I had found by hand against Crossref, and it found nothing else.

The reason this works is that checking and generating are different tasks. Asked to produce, a model reaches for what fits. Asked to verify a specific claim against a source, it has something concrete to compare against. It is the same trick as asking AI to audit its own writing for tells, applied to facts instead of style.

Give the instruction its own turn. Bolting "and check it" onto the original prompt does not work, because the model is still in the business of producing an answer.

Where it goes wrong

The source exists but doesn't say it. The most common failure now. The citation is real, the link works, and the specific claim is not in it. Only opening the page catches this.

You asked it to check itself in the same breath. One message that generates and verifies produces confident self-approval. Two messages produce an actual audit.

You read confidence as accuracy. There is no relationship between the two. A model that is 60% sure writes in exactly the same voice as one that is certain, which is why you have to ask for the uncertainty explicitly rather than listening for it.

Memory is on and quietly wrong. If an assistant stored something incorrect about you or your work weeks ago, it will keep repeating it in unrelated conversations. Worth clearing out every few weeks.

You verified the easy part. People check the headline claim and skip the numbers, names and dates hanging off it. Those are precisely where the drift happens.

You assumed a careful model is a truthful one. Not the same thing. Anthropic's own testing found Claude Opus 5 misrepresenting what it had done while running a vending machine, which is a useful reminder that capability and reliability move separately.

FAQ

What causes AI hallucinations? These systems predict what text should come next based on patterns, rather than looking anything up by default. When the pattern is strong but the specific fact is thin, the most plausible-looking detail gets filled in. That is why names, dates and figures drift more than the overall shape of an answer does.

Does turning on web search fix it? It helps a lot and does not fix it. Search grounds the answer in a real page, but the model still summarises that page, and the summary can say things the page does not. The Oumi analysis found this happening in the majority of Google's correct answers.

Which assistant makes things up least? Any answer here goes stale within weeks, so build the habit rather than picking a winner. Across every model tested, the same pattern holds: fabrication is much rarer than it was, and small errors inside true answers are not.

Is it safe to use AI for research at all? Yes, if you treat it as a fast first pass rather than a source. It is very good at finding the right material and framing the question. It is unreliable at the last mile of exact detail, which is the part you should be doing yourself anyway.

What to take from this

The instinct people have is to ask whether they can trust AI. Wrong question, because the answer changes with every release and it was never a yes or no in the first place.

The better question is narrower. Which specific claim in this answer am I about to rely on, and have I opened the page it came from? One question, asked once per task, catches most of what would otherwise get through. It is worth adding to your standing instructions so you stop having to remember it.

The error in my test was three letters long. It sat inside a correct title, a correct year and a working link, and the only thing that found it was asking the model to go back and look.