Can AI check AI’s work?

At FinDaS, AI agents build the models and reports. Every piece gets number checks run by scripts, and a review by an agent. On anything you will act on, such as a model going live or a decision memo, that review is a panel of three reviewer agents and a judge. A panel costs three reviews and a ruling instead of one review, so it is kept for the work where a mistake changes a decision.

The panel reads the work against a written brief: what the work is for and who will read it. Without a brief it does not run. A reviewer with no stated goal has nothing to measure a mistake against, so every finding looks equally urgent.

Why one AI reviewer is not enough

One reviewer is one model’s reading, with that model’s habits. Zheng and colleagues measured one of those habits in a 2023 study of AI judges. Shown the same two answers in the opposite order, the strongest judge they tested gave the same verdict both ways round in 65% of the pairs it judged. Models have improved since, so treat the figure as dated. It is still the reason the panel’s judge gets a narrow job with one written test, below: a verdict should not turn on how the work happens to be laid out.

Several judges from different model families did better than one in a 2024 preprint by Verga and colleagues. A panel of three smaller models, each from a different family, tracked human judgements more closely than one large model judging alone, carried less of any single model’s bias, and cost over seven times less. Their panel graded answers to questions, which is narrower work than reviewing a model or a memo. I take it as support for using a panel; it does not measure how well ours performs.

So the panel has three reviewer agents, each running on a different AI model, and a judge.

Why the reviewers cannot see each other’s findings

Each reviewer gets the work, the brief and the rules for rating a finding, and nothing else: no summary from whoever built the work, and none of the other reviewers’ notes.

The summary is left out because it tells a reviewer what to expect, and what to expect is the thing a reviewer should work out alone. Why not let the three compare notes? Because the second reviewer would start from the first one’s list, and two matching lists would tell you only that one was copied. Kept apart, a problem two or three reviewers found on their own is probably real.

Disagreement carries information too. If half or more of the serious findings were each found by only one reviewer, the three together have probably missed others as well, so one more reviewer reads the whole piece on its own. That extra pass happens once per piece of work.

What the judge does

The judge does not review the work from scratch. It works from the reviewers’ lists and, on a document, from the claim check’s list too.

• It groups findings that describe the same problem, so a problem is weighed once, with a note of which reviewers found it.
• It sets every finding’s severity by one test, drawn below: with this corrected, would you decide or do something different, and by how much? A real error that changes nothing you would do is high when it costs something real: a wrong figure or a claim with nothing behind it, something the brief asked for and did not get, an error your own analyst would spot and trust the rest less for. One a careful reader would notice that costs nothing is a warning. The reviewers suggest a severity; only the judge’s counts.
• It checks what can be checked. A number that can be recomputed, or a source that either says something or does not, is settled by checking it, however many reviewers raised it. Only matters of judgement are weighed, by severity and by how many reviewers found them; in the borderline band the judge rules and writes down why.
• It rules. The work fails while any critical finding stands. Otherwise it passes, and the high findings it confirms become a minimum fix list. Before the work moves on, each item is fixed, cut, or accepted as it stands by the person who signs. A finding that asks for more than the brief set, “it should also cover X”, is filed and never counts against the work.

The judge also keeps a record of its rulings, so a point settled in one review is not argued again in the next.

An example, on an invented company

Let’s take the subscription software company of our demos; it does not exist, and every figure in its memo is made up. Its decision memo recommends a retention offer: one free month for the 500 Starter customers its churn model ranks most likely to leave. The memo expects the offer to keep 45 customers who would otherwise go, each paying 10 more months at €29, so €13,050 kept against €12,000 for the offer. It recommends going ahead. The panel returns three findings.

The memo fails, on the finding only one reviewer raised. That finding could be checked: the memo values each kept customer at today’s €29, but costs the free month at the old €24. At €29 the offer costs €14,500 against €13,050 kept, a loss of €1,450, and the recommendation reverses. Counted as a vote, it would have had less support than the other real error. The test-size error had two reviewers behind it and changes nothing you would do, so it is high: a fix-list item, because your own analyst would spot it and trust the rest less. The smaller discount may well be worth testing. It was not in the brief.

Why the fix rounds stop

After the fixes, the panel does not review the whole piece again. One reviewer goes through the fix list, item by item: closed or not. It also reads everything the fixes could have broken, which means every place that repeats or relies on a changed number. In the example the cost of the offer changes, so the recommendation, the summary and any chart of the offer’s value have to change with it. The fixed memo also goes back through the number checks before that reviewer sees it.

Reviewing everything again looks safer. It is not, for two reasons. Any fresh review of a long piece of work finds something new, so a loop of full reviews has no natural end. And fixes are new work, with new mistakes in it; a check aimed at what changed looks where those are. A problem noticed outside the fixes is recorded for later, unless it would change your decision. That kind counts whenever it turns up.

If the second round fails on the same kind of problem as the first, there is no third. By then the fixing is what keeps breaking things, and the open list comes to me: I decide whether the work goes out as it stands. A second round that turns up a new kind of problem gets one more round, aimed at that.

What the panel catches, and what it misses

The panel catches work built wrong against its brief: a figure that does not recompute, a claim its source does not make, code that computes something other than what the design says, a conclusion the numbers do not support, two sections that disagree. It misses four kinds of thing.

• It measures the work against the goal it was given, so it can pass a well-built answer to the wrong question.
• It cannot know what nobody wrote down: a promotion last March, a contract about to end, a customer group your sales team treats differently. If it is in neither the data nor the brief, no reviewer sees it.
• A source it could not open, such as a paper behind a paywall, is recorded as not checked, and never counts as a pass.
• Its reviewers can share a blind spot, and using different models helps less than you would hope.

Kim and colleagues compared the errors of more than 350 AI models in a 2025 study accepted at ICML, a peer-reviewed machine learning conference. On one public leaderboard of four-option questions, when two models were both wrong they picked the same wrong answer about 60% of the time on average, where chance would give one in three. The larger, more accurate models had highly correlated errors even when different companies made them. Those were multiple-choice questions, not reviews, so the figure does not say how often three reviewers miss the same flaw. It does show that models’ misses are not independent of each other.

Why a person still signs

The panel answers one question: is this work built right against its brief? Signing answers another: should you act on it? That needs someone who knows the business, can judge whether the brief was right, and answers for the result if it goes wrong. The judge is an AI model too, with the habits measured above. So I sign what you act on, such as a model going live or a decision memo, with the panel’s findings and rulings in front of me. Routine refreshes of a signed-off report run on the automated checks and do not carry my signature.

How FinDaS does it

AI agents build the work. Scripts, not AI, run the number checks and trace cited figures to their sources. Every deliverable gets a review by an agent, and on what you will act on, that review is the panel above. Then I sign, as the analytics director. Each gate can send the work back to the agents; if two fix rounds fail on the same kind of problem, the work stops there and I decide. A fixed piece goes back through the earlier checks before it returns to the gate that sent it back. The steps are on how it works, and the offer is on the home page.

The checks you can run yourself on an AI-generated report, four of them without code, are in How to check the numbers in an AI-generated report.

Sources

1. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 Datasets and Benchmarks Track, arXiv 2306.05685.
2. Verga et al., Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models, arXiv 2404.18796, May 2024. Preprint.
3. Kim, Garg, Peng and Garg, Correlated Errors in Large Language Models, ICML 2025, arXiv 2506.07962.