Which customers will upgrade? An example propensity model readout

Recommendation. We recommend putting the model live and calling the top two tenths of each quarter’s scores, starting with today’s 805 customers, with a random part of the list left uncalled to measure what the calls add. On a quarter the model never saw, its top fifth held 62% of the customers who moved up, and its scores added up to 185 upgrades where 193 happened. The recommendation holds for customers, plans and prices like those in the history; the triggers for a refit are under “Re-checking each quarter”.

How the model was built

The model is a logistic regression on six features: the plan, tenure, seats on the account, days with a login, days at the plan’s limit and support tickets. It learned from the 3,880 customers on Starter or Growth on 31 March 2026, and from whether each moved up by 29 June 2026: 180 did, 30 for each feature tried. How many past cases a model needs is on churn prediction, where the cases are leavers; here they are upgrades.

It was then tested on the 3,955 customers of 30 June 2026, whose upgrades by 28 September 2026 it never saw. Its ranking and its scores below are judged on that quarter alone.

How well it ranks customers

The test customers are ranked by score and split into ten equal groups, tenths. Customers in the top tenth moved up at 22.0%, 4.5 times the average of 4.9%. In each of the bottom five tenths, no more than 2.5% did.

How many upgrades the list catches

Calling the top fifth of scores reaches 62% of the customers who moved up, 120 of 193; a list in random order reaches about 20%. The model’s list stays above the random one until both have called everyone.

Are the scores right?

Each score is the model’s probability that the customer moves up within 90 days. In every tenth, the predicted rate is within 1.3 percentage points of the share that moved up, and across the test quarter the model expected 185 upgrades where 193 happened, within one standard deviation of the count if the scores are right. So the scores read as probabilities: a list’s scores add up to the upgrades to expect from it.

One tenth ran outside the range the scores give at two standard deviations: 10 upgrades against 5 expected in the eighth. The other nine fell inside it. It is worth watching at the next quarterly re-check; on its own it meets none of the refit triggers below.

What moves the score

Each row compares two values of one feature for a typical customer, the other features held at that customer’s values: on Starter, with 5 seats on the account and 15 months since joining; in the last 30 days, a login on 18 days and 1 day at the plan’s usage limit; in the last 90, 1 support ticket.

The largest effect is seats on the account: 20 against 3, 3.9 times as likely to move up. Days at the plan’s limit, the signal a sales team’s usage rule reads, make moving up 3.8 times as likely at five days a month as at none. Support tickets carry little: three tickets against none come out 1.1 times as likely, within chance of no effect at all. They still count among the features tried. The effects are associations in the history, not proof of cause: whether lifting a customer’s limit or logins makes them move up is a question for a test, not for this model.

Today’s list

Ranked on the data of 30 September 2026, the top two tenths are 805 customers. Their scores add up to about 117 upgrades by 29 December 2026, more than half of the 189 the scores expect across all 4,025 customers on Starter and Growth.

Those 117 are upgrades the scores expect whether or not anyone calls. A score ranks customers; it does not show that a call causes an upgrade, and some of the list would move up on their own. We recommend leaving a random part of the list uncalled and comparing its upgrades with the called customers’ at the end of the quarter. Without that comparison, what the calls add stays unknown.

What the model is good for, and what it is not

The model is good for two things:

• choosing whom to contact first each quarter, by the chance of moving up in the next 90 days;
• the number of upgrades to expect from a list, from the sum of its scores.

It is not built for these:

• why a given customer moves up: the scores rank customers, and the effects are associations;
• what a call, an offer or a discount adds: that needs the uncalled comparison under “Today’s list”;
• customers on Scale, who have no plan to move up to, and new plans or prices the history does not hold;
• windows other than 90 days: the model was fitted and tested on that window only.

Re-checking each quarter

A propensity model drifts: customers, plans and the product change, and scores fitted on one quarter’s customers read later ones less well. In this invented history nothing changed between the three dates, so the test quarter shows no drift; on real data, each quarter’s scores are checked against who moved up, as the test quarter was here.

We recommend refitting the model when any of these happens:

• The 3,220 customers not on today’s list fall outside the upgrades their scores expect. By 29 December 2026 the scores expect 72 of them to move up: a count below 55 or above 89 is more than chance would explain, at two standard deviations. They are the check because no call reaches them.
• The uncalled part of the list moves up at no more than twice the rate of the customers not on the list. On the test quarter, the top fifth of scores moved up at 15.2% and the rest at 2.3%.
• The business changes in a way the history does not hold: a new plan, a price change, or a change in how logins or the plan’s limits are counted.
• A year has passed since the last fit.

How close the fit came to the true rule

Because the customers are synthetic, the rule that decided who moved up is known, and the fit can be held against it. On your data no such rule exists, and the test quarter is the check. Ranked by the true probabilities, the top fifth of test customers held 64% of the upgrades; ranked by the model’s scores, 62%, close to the best ranking these six features give. For 9 in 10 test customers, the model’s probability was within 1.6 percentage points of the true one. Each coefficient the fit found, the weight it gives one feature, sits within two standard errors of the rule’s own. In the table of what moves the score, the widest gap is seats on the account, 3.9 times as likely by the model against 2.7 by the rule.

Checks and sign-off

Sign-off. Putting a model live is a decision you act on, so it carries the analytics director’s sign-off. The quarterly re-scoring after that is a routine refresh: it carries its checks and no new signature. This readout is an example on an invented company and is never signed.

What it reads. Billing and plan history per customer, with the date of each plan change; product usage, such as logins and use against the plan’s limits; and support tickets, where you record them. Where the data goes is on data handling.

How it is built. AI agents rebuild your customers as they stood on past dates, fit the model and draft the readout. Scripts check its numbers and claims against the data. A model going live is something you act on, so a panel of three reviewer agents and a judge reviews it before the analytics director signs it. Each check can send the work back to be fixed; two fix rounds that fail the same way stop the work, and the director decides. The review is described on how it works.

What arrives. A readout like this one: a memo of up to two pages, with its tables as an appendix. Then the scored customer list each quarter, as a file or a recurring report, and the method note listing the checks that ran. The model and its readout are one sprint, from €12k, fixed before work starts; the quarterly re-scoring runs on an async retainer, from €8k a month.

Not covered. Calling your customers or running the sales programme, and installing anything on your systems. The model ranks whom to call; what a call adds is measured by your own comparison.