Stopping an AI From Inventing Answers

Language models are built to produce the most plausible next sentence, not the most accurate one. Most of the time those coincide. When they do not, you get a confident, well-written answer that is entirely made up — and the customer has no way to tell. Our guide to teaching an AI to sound like your business treats this as an engineering problem with known mitigations rather than a mystery.

Where invention comes from

Three situations produce nearly all of it. The first is a gap: the customer asks something your documents do not cover, and the model fills the space rather than admitting it. The second is contradiction: two documents say different things and the model blends them. The third is generalisation: the model knows what businesses like yours usually charge, and that general knowledge leaks in as if it were your price list.

All three are addressable, and none of them are fixed by telling the model to be accurate.

The four guardrails that work

1. Ground answers in retrieval. The AI should answer from your documents, not from general knowledge, for anything factual about your business — prices, policies, availability, what you do and do not offer.
2. Give it an explicit way to fail. "If the answer is not in your material, say you will check with a colleague" has to be written down, or the model will treat guessing as helpfulness.
3. Narrow the scope. An AI that only handles bookings and product questions has far fewer opportunities to invent than one that answers anything.
4. Prefer tools over memory for anything live. Availability, order status and prices should come from a lookup at the moment of asking, never from something the model absorbed earlier in the conversation.

The fourth is the one that catches teams out. If a price was mentioned twenty messages ago and has since changed, the model will happily repeat the old one. Live data belongs behind a tool call — which is part of why capability choices matter, as set out in deciding what your AI is allowed to do.

“

An AI that says "I will check" is worth more than one that is right ninety per cent of the time and never signals which ten per cent.

— be digital ai team

Test for it deliberately

You will not find hallucination by using the AI normally, because you unconsciously ask questions it can answer. Test it adversarially instead. Ask about a service you do not offer. Ask for a price for something not on your list. Ask about a policy you have never written down. Ask something factually false as a premise — "since you're open on Sundays, can I come at ten?"

In every case the correct answer is a clean admission and an offer to check. If you get a fluent invention instead, you have found a gap, and the fix is usually a document rather than a rule — see building a knowledge base your AI can actually use. Run this set of questions again after any significant change to your documents or model.

Watch the categories that hurt

Not all invented answers cost the same. A slightly wrong description of a service is a minor irritation. An invented price becomes a commitment your customer will reasonably expect you to honour. An invented delivery date turns into a complaint. An invented policy on refunds or medical suitability can turn into a legal problem.

Rank your categories and put the hardest constraints on the expensive ones. Prices and policies should be quoted only from documents, never paraphrased. Anything clinical, legal or financial should route to a person by default, which is exactly the sort of trigger described in when the AI should stop and fetch a human.

When it has already happened

Assume it will, and decide in advance how you respond. Generally, honour the mistake if it is small and correct it if it is not — a customer quoted twenty euros too little should get the twenty-euro price, because arguing costs you more than the difference. Where the gap is large, a person should call, explain plainly that the automated reply was wrong, and offer something for the inconvenience.

Then treat the incident as information. Every invented answer points to a missing document or an over-broad scope, and the audit log will show you exactly what the AI had in front of it when it went wrong — the review habit covered in reading the AI audit log is what turns a bad week into a permanently narrower failure surface.