A language model produces the most plausible continuation of the text so far. It has no concept of not knowing. Left alone, “what are your opening hours?” gets opening hours, invented, delivered with total confidence. Everything below is about removing the conditions that let that happen.
Cause 1: nothing was retrieved
The most common cause. The question found no relevant passage — the page was never indexed, the extraction failed, or the wording did not match — and the model filled the gap.
Fix: make an empty retrieval explicit rather than silent. The instruction should say: you have no reference information, so say you do not have it. That one sentence removes most invented answers.
Cause 2: the wrong passage won
The answer is confident and sourced, but sourced from the wrong place — an old page, a blog post, a different product.
Fix: remove duplicates, keep canonical facts in one authoritative place that overrides pages, and mark blog posts as background so they cannot become claims about your services.
Cause 3: the model was asked to be helpful
Prompts that say “always be helpful and provide a complete answer” are instructions to keep talking when the honest reply is one sentence. Helpfulness and accuracy pull against each other.
Fix: say explicitly that declining is an acceptable, good answer.
Cause 4: facts the model can generate
Prices, phone numbers, dates and URLs are the dangerous class, because they are exactly what a model can produce convincingly. A made-up link to a page that does not exist is the classic.
Fix: never let the model construct these. Prices come from the shop database. Links come from a list of real pages it was given. If it was not given one, it cannot produce one.
How to tell whether a tool takes this seriously
- Ask it something your site does not cover. A good one says it does not know. A bad one improvises.
- Ask for a link. Check the link exists.
- Ask about a price. Change the price. Ask again.
- Look for a screen that shows which passages produced an answer.
You cannot make a language model incapable of inventing. What you can do is remove the situations where inventing is its best available move: retrieve properly, make “I do not know” acceptable, and take the facts it could fabricate out of its hands entirely.
Questions
Does a bigger model hallucinate less?
Somewhat, but far less than better retrieval does. A small model with the right passage in front of it beats a flagship model guessing.
Can I get it to zero?
No. You can get it rare, make failures honest, and catch the rest with the unanswered-questions report.