One customer view across online and in-store — with an AI analyst that cannot invent a number

The choice

The analyst is given fourteen specific, pre-built, tested data functions and can call those and nothing else — at most fourteen times, at most three levels deep. There is no facility anywhere in the system for executing a model-written query.

Why

The obvious way to build an AI analyst is to let the model write SQL. It is also the way that produces a system nobody can trust, because the failure modes are unbounded: a wrong join, a wrong date window, a query that runs for nine minutes, or a perfectly valid query that answers a subtly different question than the one asked.

What it means for you

A dangerous query is not caught and blocked — it cannot be composed in the first place. Injection is unreachable rather than defended against.

The choice

Incoming questions are matched against 85 questions written and checked by hand, using word stemming, a synonym map, and a scoring gate with required and bonus terms. There is no vector database and no embedding search anywhere in the system.

Why

Nearest-neighbour retrieval always returns something. When a question is unlike anything it has seen, it returns its most confident wrong answer, and nothing about that is visible to the person who asked. Exact rules can fail loudly instead.

What it means for you

When the system recognises your question you are getting an answer a human has already verified. When it does not, it says so and works the question out with tools — where every number still has to survive the grounding check. The cost is that the question bank is maintained by hand. That is the trade, made deliberately.

The choice

Malformed tool arguments are repaired rather than silently mishandled. A filter that is not on the approved list of 78 names is a hard error, not a best-effort guess. Every number in the written answer is extracted and checked against the data actually fetched. And where a user is restricted to certain stores, any tool that cannot honour that restriction is removed from the model’s options entirely, with the finished answer re-checked before it is shown.

Why

These are four genuinely different ways to be wrong, and a single safeguard catches only one of them. The dangerous one is the third: fluent prose with a plausible figure that appears nowhere in your data.

What it means for you

If a figure is not in the fetched results, the whole answer is discarded and replaced by one assembled directly from the data — an answer that is structurally incapable of containing an invented number. On a detected scope leak, the response is refused rather than trimmed.

The choice

A suite re-asks a fixed set of questions and checks three things: was the right date window used, is every number traceable to fetched data, and does the result agree with the dashboard’s own endpoint for the same metric.

Why

An answer is only worth trusting if it has been tested.

What it means for you

The current golden set passes 153 of 153 across six suites. Alongside it, 52 data checks guard the warehouse itself — including the one most pipelines skip: an assertion that a closed month’s numbers are still the same after tonight’s refresh.