A wrong answer delivered hesitantly is a nuisance. A wrong answer delivered confidently, with no way to check it, is a liability. Most AI software in use today produces the second kind, and the industry has decided to call this an accuracy problem. It is a design problem.
The first time it costs something
Here is the sequence, and it is remarkably consistent.
Someone asks the system a question. It answers with a number. The number is wrong, or right about the wrong period, or right for one entity when the person meant the group. Nothing about the reply signals this, because language models are fluent whether or not they are correct, and fluency reads as confidence.
The person acts on it. They quote a customer, or hold a payment, or tell the MD something in a meeting. Two weeks later it surfaces.
What happens next is the interesting part. The company does not conclude that this particular answer was wrong. It concludes that the system cannot be trusted, and that conclusion is permanent and total. One incident is enough. The tool is now a thing people mention with a small laugh.
You do not get to explain that it is right ninety-four percent of the time. Ninety-four percent is a terrible number for anything a person will act on, and everybody in the room understands that instinctively even if nobody says it.
A verified tool survives. A trusted one gets exactly one chance.
Accuracy is the wrong target
The instinct is to chase the last few percent. Better model, better retrieval, more evaluation. All worth doing, and none of it solves this, because the residual error rate never reaches zero and the person still has no way to tell which answer they are looking at.
The target is not a system that is never wrong. It is a system where being wrong is visible in under ten seconds.
Those are completely different design problems, and only one of them is achievable.
What a checkable answer looks like
Take the same question two ways.
Which dealers are overdue beyond sixty days?
Answer one: "Twelve dealers are currently overdue beyond sixty days, totalling approximately ₹47 lakh."
Answer two: the same sentence, followed by the twelve names with amounts and ageing, each linked to the ledger entry, with a line saying which report this was computed from and as of what date.
The first is a claim. The second is evidence. And the crucial thing is what happens when the second one is wrong: the finance head glances at the list, sees a name that was settled last Thursday, and knows immediately that the ledger sync is stale. Thirty seconds, and the system's credibility survives, because it did not ask to be believed. It showed its work.
This is the difference between a tool people trust and a tool people verify. Verified tools survive. Trusted ones get one chance.
Four rules we build to
Cite at the atom, not the document
"Source: Sales Report 2025.xlsx" is nearly useless in a file with forty thousand rows. The citation has to be the row, the voucher, the paragraph. If a person has to search inside the source to find what you meant, they will do the work manually instead, and then twice.
Say when the data is from
Most wrong answers in business AI are not hallucinations. They are correct answers to a slightly different question, usually about a different period or a different entity. Stating the as-of date and the scope on every reply removes a large share of the error class before anyone notices it existed.
Refusing is a feature
"I do not have this. The last purchase data I can see ends in March" is a good answer. It is more useful than a plausible guess, and it tells you something true about your own systems.
Building this in takes deliberate effort, because the underlying models would rather be helpful than accurate. It is worth the effort. A system that admits its edges is one people learn to rely on inside those edges.
Show the question it thought you asked
When someone asks about overdue dealers, the system made several decisions: which entity, which date range, whether to count disputed invoices. Surfacing those quietly under the answer lets the person correct the interpretation rather than the number, which is faster and teaches them how to ask better next time.
What belongs on every answer
- The row, voucher or paragraph it came from, not just the file name
- The as-of date and the entity or period in scope
- A plain refusal where the data runs out, rather than a plausible guess
- The question it understood you to be asking
Why this is a design job
Every one of those four rules is an interface decision, not a model decision. Where the citation sits, how much of it shows before you click, whether the as-of date is a caption or a warning, how a refusal is worded so it reads as competence rather than failure.
Get that wrong and a technically excellent system is unusable. A reply that dumps forty citations in a wall of grey text is not verifiable, it is just noisy, and people skip straight to the bold number, which is precisely the failure you were trying to prevent.
This is the part we think most AI companies are structurally bad at, and not through carelessness. They are engineering teams who add an interface at the end. We came from ten years of studio work, where the whole discipline is the gap between a thing being correct and a person being willing to act on it. Coming to AI from that direction turns out to be an advantage, because the hard remaining problem in this field is no longer the model.
The question to ask a vendor
One question, and it separates demos very quickly.
Show me an answer, then show me how I check it.
If the second half takes more than a few seconds, or involves opening another system, or the answer is that the model is very accurate, you have learned what you needed to know. Related reading: most AI projects do not fail, they go unused, which is largely the downstream consequence of getting this wrong.
