Churn prediction for companies without a data team

A 1996 simulation study of logistic regression, a standard model for a yes-or-no outcome such as leaving, found no major problems with 10 or more events per variable in the model; below 10, the model’s estimates were biased. Here an event is a customer who left, and a variable is a signal such as logins, seats in use, support tickets or failed payments. We count every signal the work tries, not only the ones the final model keeps: trying many and keeping the best draws on the same leavers, so this count asks for more data than the study’s, never less.

So ten leavers per signal is the floor. A 2014 simulation study found that logistic regression ranked cases stably only from about 20 to 50 events per predictor, and that random forests, support vector machines and neural networks may need over ten times as many: they were still unstable above 200. Both studies simulated medical outcomes, not churn, so read their figures as orders of magnitude.

Two more limits apply whatever the count. If your business has seasons, the history has to cover at least a full year, or a seasonal dip looks like a lasting one. And the latest months are held back to test the model, so they do not count towards training.

With too few leavers, the right first deliverable is smaller: an analysis of who left and when, which needs no model and says what to record until there is enough history for one. A figure built on a handful of records moves with whichever records happen to be in it; one of our own reports showed how far.

The proposal follows a free call of up to thirty minutes.

Repeat work after the model is live runs on an async retainer, from €8k a month.

Who it is for. Companies whose revenue repeats, through subscriptions, contracts or repeat purchases, and who have no data team of their own.

The questions it answers. Which customers are likely to leave in the next month or quarter. Which signals come before leaving. How much revenue is at risk, and which customers to contact first. What a customer is worth once you know how long they stay is on the customer value page.

What you need to have. Most of it sits in a data warehouse or in your billing system’s exports:

• a customer list with start dates, and end dates for those who left;
• billing and plan history;
• product usage, if you record it;
• support tickets and contact history, where they exist.

The method, in plain words.

1. Define leaving precisely: a cancellation, or no renewal within an agreed number of days.
2. Rebuild what was known about each customer at past dates, using only what was known then, and whether they left in the window that followed.
3. Train the model on the older months and test it on the latest ones, which it never saw.
4. Compare it with a simple rule you could run without a model, such as “no login for 30 days”. A model that cannot beat the rule is not worth running.
5. Show which signals drive the scores, so the list can be acted on.

What you receive. A decision memo of up to two pages: how well the model ranks customers on the months it never saw, how it compares with the simple rule, and what to do with the scores. The scored customer list, as a file or a recurring report. A method note listing the checks that ran. An example readout, on an invented company, shows what the memo reports.

How it is checked. Agents build the model and draft the memo. Every deliverable gets number and claim checks run by scripts, not AI, and a review by an agent. On what you act on, such as a model going live or a decision memo, that review is a panel of three reviewer agents and a judge, and then the analytics director signs it; routine refreshes do not need a sign-off. The checks are described on how it works.

Your data. Where it goes, which model providers process it and what happens when the work ends: see data handling.

Who builds it. FinDaS AI’s analytics director built churn, propensity and next-best-offer models during eight years in retail-banking and consumer-finance CRM analytics, where he also built two analytics teams. Churn is one of the money questions the same team works on.

A subscription software company has 4,000 paying customers at €77 a month on average and loses about 4% of them a month: about 160 leavers. Sign-ups more than replace them, so the base grows slowly; to keep the arithmetic simple, the example counts 4,000 customers and 160 leavers in every month. It has 24 months of history and 30 candidate signals.

The company holds back its latest 6 months for the test and trains on the 18 before them: 2,880 leavers, 96 per signal. That is enough for logistic regression and too few for the flexible models.

On the 6 test months, the model’s top tenth held 56 of each month’s 160 leavers, against 32 for the simple rule, so the model earns its place. If a retention call keeps 1 in 5 of the leavers it reaches, the model’s list keeps about 11 customers a month, worth €847 in monthly revenue. What a lasting cut in churn is worth over two years, and how sure that is, is the question of our revenue forecast demo, on the same company. The arithmetic behind that question, one step at a time, is in What is reducing churn by 1% worth?

A company with 800 customers would lose 32 a month and need about 10 months of history to reach even the floor of 300 leavers. Its first deliverable would be the analysis of who left.