Can AI replace a data science team?

The evidence splits along one line. On questions someone defined in advance, with the data shaped to answer them, the AI models in dbt Labs’ test answered 90% or more correctly through its semantic layer. On open-ended analysis the best AI agents fall well short of skilled people, and their mistakes arrive without an error message. A data science team is hired for both kinds of work, and AI covers the first far better than the second.

On paper, the job is not shrinking. The US Bureau of Labor Statistics projects employment of data scientists to grow 35% from 2025 to 2035, from about 275,600 jobs in 2025.

Everything below is about analysis and modelling. Data engineering, the pipelines and warehouses that analysis runs on, is a separate question, and I leave it aside.

Where AI is already good enough

The strongest case for AI comes from dbt Labs, which sells the semantic layer it tested (so, a vendor grading its own product). dbt took 11 business questions about a sample insurance dataset and had an AI build the data models needed to answer them: three new models, with no code written by people. Asked each question 20 times, Claude Sonnet 4.6 answered 98.2% correctly through the semantic layer and 90.0% by writing SQL directly. On the same data before that modelling, it scored 72.7% and 64.5%.

The caveat? Two conditions carry that result. The questions were known before the work started, and the data was modelled to answer exactly those questions. dbt’s own write-up says that questions the layer could not answer in 2023 still cannot be answered without more modelling. Its argument for the layer is that it says when it cannot answer, rather than returning a wrong number. That is dbt’s argument; the benchmark did not measure it.

On competition tasks with a fixed scoring rule, agents working alone already beat most teams of people. AgentDS, a 2025 competition of 17 data science challenges across six industries (reported in a 2026 preprint), set 29 teams of people, free to use any AI tools, against AI agents working with no human input and ten minutes per challenge. The best agent ranked 9th of the 29 teams, above the median team.

My read: a routine refresh, the same defined report on new data each month, is work an agent can carry with automated checks.

Where it falls short: the open questions

Open-ended work looks different. DSAgentBench, a 2026 benchmark of 275 tasks covering the whole data science workflow (a preprint, not yet peer-reviewed), counts a task as solved when its output scores at least 0.95 on the benchmark’s checks. The strongest agent tested, Claude Sonnet 4.6, solved 56.7%. Three people, two applied scientists and a master’s graduate, solved 85.1% under the same conditions. A weaker model? No, the same one that scored 98.2% on dbt’s set questions; what changed is the kind of question.

The gap at the top of AgentDS was just as wide. The best team of people working with AI scored 0.860 on the competition’s 0-to-1 ranking scale; the best agent working alone scored 0.510.

Real questions show the same pattern, and how AI fails matters more than how often. ClickHouse took 201 real questions from the traffic of its own internal analytics agent. The best model, Claude Fable 5.1, answered 76.6% correctly. Runtime errors were close to absent, so the wrong answers came back as clean results. For every model the most common failure was a wrong plan for answering the question, not broken code. The answer key, though, was set wherever two of three other AI models agreed, and questions where all three differed were dropped. So the benchmark leans towards questions with one clear answer, and the real error rate on the harder ones is probably higher.

A wrong plan is exactly the mistake it takes someone who knows the business to catch: churn measured on the wrong customer base, a promotion month compared with an ordinary one, revenue counted when it was invoiced rather than when it was earned. In each of these the query runs perfectly.

And the same analysis can come out both ways. A study in PNAS in July 2026 had AI agents analyse the same three datasets and hypotheses independently, 4,946 times. Independent runs often reached opposite verdicts on whether the hypothesis held. Framing alone moved them: between agents told the hypothesis was implausible and agents pushed to find support for it, the share of runs judging it supported differed by 34 to 66 percentage points, depending on the dataset.

How to tell which kind of question you have

A question is defined when its metric is fixed, you know where its data comes from, and last month’s answer tells you roughly what this month’s should look like. Revenue by plan is one; active customers by region is another. AI agents handle these well once someone has set them up, and a wrong answer usually stands out against the history.

A question is open when choosing the method is part of the answer: why churn rose last quarter, which customers to offer a discount, what a price rise would do to sign-ups. There is no last month to compare with. Several reasonable methods give different answers, and the wrong one looks just as finished as the right one. This is the work a data scientist is hired for, and it is where the benchmarks above show AI falling short.

What it means if you have no data team

The usual answer to AI’s error rate is to have a person check its work. Fine, but the evidence says that person has to be good at the task. A 2024 meta-analysis in Nature Human Behaviour pooled 106 experimental studies, with 370 results, comparing people working with AI against people or AI working alone. On average the pair did worse than the better of the two alone. Where the person was better at the task than the AI, the pair gained; where the AI was better, the pair lost. Pairs also tended to lose on tasks that end in a decision, and to gain on tasks that create something.

Warning people does not close the gap either. In a 2025 study (a preprint), marketing, sales and business development professionals reviewed AI analyses of marketing data, each with at least one flaw the authors had identified in the AI’s response, such as ignoring seasonality. Every task told them the AI’s methods often contain mistakes. They still missed flaws that mattered, and some flaws no participant caught. The study is small, 10 people in its first round and 18 in its second, so I would treat it as directional.

A company without a data team has nobody in that seat. That leaves four workable options, and each has a cost:

• Keep AI on questions defined in advance, on data modelled to answer them, which is where the dbt result applies. Someone still has to define the questions and model the data, and anything new falls outside it.
• Accept the error rate on decisions that are cheap to reverse. It is the cheapest option, and the riskiest the first time a decision turns out not to be.
• Hire a data scientist and let AI speed up their work. It costs a salary, and one person is a team of one.
• Bring in a senior from outside to run the agents and answer for what they produce. You depend on someone outside the company, and their judgement is only as good as their access to your context.

How FinDaS does it

FinDaS takes the fourth option, so I am obviously not neutral about the list above. AI agents build the models and reports, a review round checks them, and each deliverable carries a short note of which checks ran. I built two analytics teams before this one, and I sign what you act on, such as a model going live or a decision memo. Routine refreshes of a signed-off report run on the agents’ checks. The steps are on how it works.

If you already receive AI-generated reports, there are six checks to run on their numbers, four of them without reading code: How to check the numbers in an AI-generated report.

Sources

1. dbt Labs (Jason Ganz, Benoit Perigaud), Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update, 7 April 2026. A vendor test of its own product.
2. Luo et al., AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science, arXiv 2603.19005, version of 3 June 2026. Preprint.
3. Rahman et al., DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?, arXiv 2608.10366, 11 August 2026. Preprint.
4. ClickHouse (Eduardo Vellasques, Al Brown), The Agentic Analytics Benchmark, September 2026. A practitioner benchmark with a model-voted answer key.
5. Bertran, Fogliato and Wu, Many AI analysts, one dataset: Navigating the agentic data science multiverse, PNAS 123(29), July 2026.
6. Vaccaro, Almaatouq and Malone, When combinations of humans and AI are useful: A systematic review and meta-analysis, Nature Human Behaviour, 2024.
7. Virk and Liu, Non-programmers Assessing AI-Generated Code: A Case Study of Business Users Analyzing Data, arXiv 2508.06484, August 2025. Preprint.
8. US Bureau of Labor Statistics, Occupational Outlook Handbook: Data Scientists, last modified 27 August 2026.