How to tell if a churn model works: an example readout

Recommendation. We recommend putting the model live and calling the top tenth of its scores each month, about 437 customers. On months it never saw, it reached 1.6 times the simple rule’s leavers on the same number of calls, and its scores matched the share who left. The recommendation holds for customers, plans and prices like those in the history; the triggers for a refit are under “When to refit”.

Is there enough history?

A churn model learns from the customers who left, so the count that matters is leavers per signal tried. The training months, October 2024 to March 2026, hold 2,700 leavers, and we tried 8 signals: days with a login, seats in use, support tickets, failed payments, the Growth plan, the Scale plan, tenure and seats on the account. That is 338 leavers per signal. A 1996 simulation study of logistic regression, the model used here, found no major problems from 10 events per variable, and a 2014 one found that logistic regression ranked cases stably from about 20 to 50 events per predictor; both simulated medical outcomes, not churn. This history clears both figures several times over. How the count works, and what to do below it, is on churn prediction.

How well it ranks customers

Pick one customer who left in the test months and one who stayed: the model gave the leaver the higher score in 74 pairs of 100. A coin scores 50, and the simple rule 56. On the training months the figure is 74, so the model ranks the months it never saw about as well as those it learned from.

Ranked each month, the top tenth of scores, about 437 customers, left at 13.7% a month against 3.8% for all customers: 3.6 times the average.

Against a simple rule

The rule “no login for 30 days” needs no model. Over the 6 test months it flagged 1,610 customers, about 268 a month, and 169 of them left. Calling the model’s highest scores instead, the same number each month, reaches 263 leavers: 1.6 times as many for the same calls. A model that cannot beat the rule is not worth running; this one beats it in every three-month stretch of the test.

Are the scores right?

Each score is the model’s probability that the customer leaves in the next month. In every tenth of scores, the predicted rate is within 1.0 percentage points of the share that left, and across the 6 months the model expected 980 leavers where 997 left. So the scores read as probabilities: a list’s scores add up to the leavers to expect from it.

What drives the scores

Each row compares two values of one signal for a typical customer, the other signals held at that customer’s values: on Starter, 8 seats on the account, 20 months since joining, a login on 17 days of the month, 75% of seats in use, no support tickets and no failed payments.

Failed payments carry the most: one in a month makes leaving the next month 3.4 times as likely. Seats on the account carry almost nothing: a 3-seat account and a 30-seat one leave at about the same rate. They still count among the signals tried. The effects are associations in the history, not proof of cause: whether getting a customer to log in more keeps them is a question for a test, not for this model.

What the list is worth

Over the 6 test months, calling the top tenth each month reaches 360 leavers, who pay €11,630 a month between them. If a call keeps 1 in 5 of the leavers it reaches, the calls keep about 72 customers and €2,326 of monthly revenue. The rule’s list, on the same assumption, keeps about 34 customers and €1,436.

The 1 in 5 is an assumption, not a finding. Not priced here: what the calls cost. What a lasting cut in churn is worth over two years is the question of our revenue forecast demo, on the same invented company.

What the model is good for, and what it is not

The model is good for two things:

• choosing whom to contact first each month, by the chance of leaving in the next month;
• the number of leavers to expect next month, from the sum of the scores.

It is not built for these:

• why a given customer leaves: the scores rank risk, and the effects are associations;
• leaving further ahead than a month: the model was fitted and tested on next-month leaving only;
• customers, plans or prices the history does not hold, such as a new plan or a price change;
• whether a call keeps anyone: that needs the comparison under “What to do with the list”.

What to do with the list

We recommend this routine each month:

1. Score every paying customer on the month’s data, at the end of the month.
2. Call the top tenth, about 437 customers. Send customers with a failed payment to billing first, since a card update fixes a failed payment and a retention offer does not.
3. Leave a random part of the list uncalled, and compare its leaving with the called customers’ two months on. Without that comparison, what the calls keep stays an assumption, like the 1 in 5 under “What the list is worth”.
4. Count each month’s leavers against the sum of the scores; the refit triggers read that count.

When to refit

We recommend refitting the model when any of these happens:

• Over three months, the model’s list reaches no more leavers than the rule on the same number of calls.
• Over three months, the leavers fall outside the range the scores expect. For July to September 2026, the model expected 498: a count below 455 or above 541 is more than chance would explain, at two standard deviations. Every three-month stretch of the test fell inside its range.
• The business changes in a way the history does not hold: a price change, a new plan, or a change in how logins or seats are counted.
• A year has passed since the last fit. The refit reads the latest 24 months, so the training months keep a full year of seasons.

Checks and sign-off

Sign-off. Putting a model live is a decision you act on, so it carries the analytics director’s sign-off. The monthly re-scoring after that is a routine refresh: it carries its checks and no new signature. This readout is an example on an invented company and is never signed.

What it reads. A customer list with start dates, and end dates for those who left; billing and plan history; product usage and support tickets, where you record them. How many leavers a model needs is on churn prediction, and where the data goes is on data handling.

How it is built. AI agents rebuild your customer history month by month, fit the model and draft the readout. Scripts, not AI, check its numbers and claims, and before the model goes live a panel of three reviewer agents and a judge reviews it. Then the analytics director signs it.

What arrives. A readout like this one: a memo of up to two pages, with its tables as an appendix. Then the scored customer list, as a file or a monthly report, and the method note listing the checks that ran. The model and its readout are one sprint, from €12k, fixed before work starts; the monthly re-scoring runs on an async retainer, from €8k a month.

Not covered. Calling your customers or running the retention programme, and installing anything on your systems. The model ranks who to call; what a call keeps is measured by your own comparison.

How many leavers the list reaches at the number of calls your team can make, against the simple rule’s, is in the who-to-call demo, on the same invented company.