Six checks, in the order I would run them:
- Trace each number to its source query or document.
- Compare it against a total you already trust.
- Check the denominator.
- Read the SQL.
- Ask for the same answer twice.
- Re-run it on a second sample.
Checks 1, 2, 3 and 5 need no code, so whoever owns the decision can run them. Checks 4 and 6 need someone who reads SQL and knows when a pattern is real. For the worked examples, let’s take the two invented companies of our demos: a subscription software company with 4,000 paying customers on plans at €29, €99 and €299 a month, and, for the one question about orders, an online coffee shop. Neither exists, and every figure in the examples is made up for this article.
These checks catch numbers that are wrong. They will not tell you whether the report asked the right question in the first place; that takes someone who knows both the business and the method.
Why reading is not enough
The US National Institute of Standards and Technology names the failure confabulation: generative AI producing confidently stated content that is false. Its 2024 profile of generative AI risks adds that a model can set out logical-looking steps for an answer that is itself wrong. So a plausible explanation is no evidence that the number is right.
A wrong number does not come with a warning either. In a 2026 test by ClickHouse on 201 real analytics questions, the best model answered 76.6% correctly, measured against an answer key that other AI models agreed on. Runtime errors were close to absent: the wrong answers came back as clean results.
Warning people in advance does not help much. In a small 2025 study (a preprint, 10 people in its first round and 18 in its second), business professionals were told on every task that the AI’s methods often contain mistakes. They still missed flaws that mattered.
So none of the checks below judges the report’s reasoning. They test its numbers against things outside the report, the way the US Government Accountability Office’s guide, Assessing Data Reliability, tests any data an audit relies on: compare the record count with totals held elsewhere, and trace a sample of records to their source documents.
The six checks
Each check tests a different place a number can go wrong, and four of them need no code
1. Trace each number to its source query or document
Every figure a decision rests on should point to the query that produced it, or to the page of the document it came from. Ask for that link, number by number. NIST’s profile lists reviewing the sources and citations in AI output among its suggested actions. A number nobody can trace is an estimate at best, and should be labelled as one.
Worked example. Let’s say you asked how fast recurring revenue grew. The report says 10.1% in the third quarter, from €1,000,000 to €1,100,730. The query behind it sums invoices by invoice date. Traced back, July holds €107,640 from 30 Scale customers who moved to paying a year up front, each year invoiced in a single month. Spread over the twelve months they pay for, those invoices add €26,910 to the quarter, not €107,640: the quarter comes to €1,020,000, and recurring revenue grew 2.0%.
Spread over the months they pay for, the July invoices add €26,910 to the quarter, not €107,640
2. Compare it against a total you already trust
Before reading what a report says about a group, check that the whole group is there. Find how many rows the analysis used and hold that count against a figure from another system, such as customers in billing or revenue in the accounts. The GAO guide’s data tests include the same one: the number of records checked against the agency’s own totals.
Worked example. A churn analysis reports on 3,600 customers. The billing system shows 4,000 active at the start of the quarter. The missing 400 are the customers on the Scale plan, billed by invoice and held in a separate table the query never joined. Every rate in the analysis describes 90% of the customer base, and the 10% it left out bring in 39% of the revenue and leave least often.
3. Check the denominator
A rate is two numbers, and a report usually prints only the result. Ask what was divided by what, and whether the bottom number means the same thing from one period to the next.
Worked example. The report says monthly churn fell from 4.0% to 3.0%. Last month it divided 160 cancellations by the 4,000 customers active at the start of the month. This month it divided 150 cancellations by 5,000: every customer active at any point in the month, including 1,000 who joined in a promotion. On last month’s definition, this month is 150 out of 4,000, or 3.75%: a fall of a quarter of a point, not a whole one.
On last month's definition, churn fell a quarter of a point, not a whole one
The same data as a table
| Month and definition | Cancellations | Divided by | Churn |
|---|---|---|---|
| Last month, active at the start | 160 | 4,000 | 4.0% |
| This month, active at any point | 150 | 5,000 | 3.0% |
| This month, active at the start | 150 | 4,000 | 3.75% |
A version of this actually happened, on my own company’s site. GA4 filed nine in ten visits to it as direct, and only 9% of those direct visits engaged. Screen sizes showed that most of them were browser automation. Once the machines had a category of their own they stopped diluting the rate, and the direct visits left engaged at 20%. The 9% was correct arithmetic on a denominator made mostly of machines. How we found it.
4. Read the SQL
A query that runs is not a query that is right. Read it, or have someone who reads SQL do it, and look in four places: the joins, which can multiply rows; the filters, which decide what is left out; the date window, including where it starts and ends; and how missing values are treated.
Worked example. The coffee shop’s report gives an average order value of €65. The query joins orders to order lines, one row for each product in an order, and then averages the order total across the joined rows. An order with three products counts three times, and large orders, which have the most lines, are counted most. Averaged once per order, as the question meant, the figure is €40.
Joined to its lines, an order counts once for every product in it
5. Ask for the same answer twice
Put the same question to a fresh session, worded neutrally, and compare the answers. In a 2026 study in PNAS of 4,946 independent AI analyses of the same datasets, runs often reached opposite verdicts on the same hypothesis. Wording mattered too: agents pushed to find support judged the hypothesis supported 34 to 66 percentage points more often than agents told it was implausible. Ask “did sign-ups change after the price rise?” Asking it to “show how the price rise hurt sign-ups” decides the answer before the analysis starts. If two runs disagree, the analysis is not settled, whichever of them turns out to be right.
Worked example. You ask whether the March price rise cut sign-ups. The first run says yes: sign-ups fell 12% against February. A second, fresh run compares March with March a year earlier and finds no change, because sign-ups dip every March. The second run is the better analysis, and a single run could have given you either one.
6. Re-run it on a second sample
A pattern found in one slice of data is a lead. It becomes a finding when it holds on data the analysis has not seen, so ask for it to be run again on a later period, another region, or a random part of the data held back from the start. If the effect shrinks or disappears, it was most likely noise, or a one-off event in the first sample.
Worked example. An analysis of January to June finds that customers who contact support twice in their first month churn at three times the rate of everyone else: 12% a month against 4%. Re-run on July to September, the same group churns at 5% against 4%. Most of the first gap came from one outage week in June, which produced both the support tickets and the cancellations.
On months the analysis had not seen, the gap all but closed
January to June: where the pattern was found
July to September: the second sample
The same data as a table
| Period | Group | Churn a month |
|---|---|---|
| January to June | Contacted support twice | 12% |
| January to June | Everyone else | 4% |
| July to September | Contacted support twice | 5% |
| July to September | Everyone else | 4% |
What it means if you have no data team
Anyone who owns a decision can run four of the six checks without code: ask for the source of each number, hold the row count against billing, ask what a rate was divided by, and run the question twice. The other two, reading the SQL and judging whether a pattern survives a second sample, need someone who could have done the analysis themselves. Warning a non-specialist about mistakes is no substitute (the study above tried exactly that).
How deep to go depends on what rides on the number. The GAO guide asks for a more thorough assessment wherever bad data would do more damage to the decisions built on it. I would give a figure on a weekly dashboard the first three checks, and a number behind a price change, a hire or a model going live all six, from someone who could have produced it.
How FinDaS does it
At FinDaS, AI agents build the analysis. Every deliverable gets number and claim checks run by scripts, not AI, and a review by an agent. Two of the six checks above can run automatically there: a script traces cited figures to their sources, and a review agent reads the code before a number ships. On what you act on, such as a model going live or a decision memo, the review is a panel of three reviewer agents and a judge, and then I sign it. Each deliverable carries a short note saying which checks ran on it, and the judgement calls are mine. The steps are on how it works.
Sources
- National Institute of Standards and Technology, NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024.
- US Government Accountability Office, GAO-20-283G, Assessing Data Reliability, December 2019.
- ClickHouse (Eduardo Vellasques, Al Brown), The Agentic Analytics Benchmark, September 2026. A practitioner benchmark with a model-voted answer key.
- Virk and Liu, Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data, arXiv 2508.06484, August 2025. Preprint.
- Bertran, Fogliato and Wu, Many AI analysts, one dataset: Navigating the agentic data science multiverse, PNAS 123(29), July 2026.