Three numbers set the size of a price test: how often the thing you measure happens today, the smallest change worth detecting, and how sure you want to be.
For a rate, such as the share of sign-ups who start paying or the share of customers who leave, the standard formula (the normal approximation for comparing two proportions) gives the customers needed in each group:
n = (1.96 + 0.84)² × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ gap²
Here p₁ and p₂ are the true rates in the two groups, at today’s price and at the new one, and the gap is the distance the test’s range has to clear. To see a change, the gap is the change itself. To show that a change sits below a line, the gap is the distance from the change to the line; if the price changes nothing, p₂ equals p₁ and the gap is the whole distance to the line. The 1.96 sets 95% confidence. The 0.84 sets 80% power: when the truth is as assumed, a test of this size clears the gap in 8 tests of 10. Both are rounded here, and the figures on this page use the unrounded values. The formula assumes two groups of equal size, chosen at random. The price-test calculator runs the same arithmetic on your own numbers.
Both settings are conventions. ICH E9, the guideline on statistics in clinical trials written for drug regulators in the EU, Japan and the US, says the chance of a false alarm is conventionally set at 5% or less, and the chance of missing a real effect at 10% to 20%. It also says other values may be acceptable, or even preferable, in some cases.
The smallest effect worth detecting is your break-even
For a price rise, the change worth detecting is the one that turns the rise from a gain into a loss. A rise of r on a margin of m, both as shares of today’s price, holds total margin while customers fall by less than r ÷ (m + r): for a 10% rise at an 80% margin, 11.1%. In a test, a customer who would have left anyway, or who would never have started paying, pays neither price, so the line applies to the rest: 11.1% of the customers who would have stayed, or of the sign-ups who would have paid. The pricing analysis page works the line through. Drug trials make the same move: E9 lets a trial base the difference it must detect on the smallest effect that would matter to patients.
At 11.1% of customers lost, what the rise adds equals what the leavers took
Where that line sits against today’s rate decides the size of the test. And the same price rise puts it in very different places.
One price rise, three questions
Let’s take the invented subscription company from the pricing analysis page and its entry plan, Starter: 2,400 customers at €29 a month, an 80% margin, and 5% of them leaving each month. The company gets 600 trial sign-ups a month, and 25% of them start paying, on Starter. It is considering €31.90. The company does not exist; every figure below is made up, and computed when this page is built.
The same rise puts the break-even 2.8 points from today’s rate for sign-ups and 9.5 for the first wave: 3,815 customers a group against 213
The same data as a table
| What the test measures | Rate today | Rate at the break-even | Customers a group |
|---|---|---|---|
| Sign-ups who start paying | 25.0% | 22.2% | 3,815 |
| Customers leaving within three months | 14.3% | 23.8% | 213 |
| Customers leaving within a year, if the rise lifts churn for good | 46.0% | 50.1% | 2,303 |
Sign-ups: more than a year of them
A new customer at €31.90 brings €26.10 of margin a month, against €23.20 at €29, so the rise pays as long as conversion stays above 22.2%. If the price changes nothing, showing that conversion stays above that line takes 3,815 sign-ups in each group, 7,630 in all: at 600 a month, 13 months of sign-ups. A three-month test, with 900 sign-ups a group, could reliably see conversion falling only as far as about 19.5%. That is more than a fifth of paying sign-ups lost, about twice the break-even.
The first wave of existing customers: 213 a group
For existing customers the line sits somewhere else entirely. Over three months, 14.3% of them leave at the old price anyway. The rise pays as long as the extra leaving stays under 11.1% of the 85.7% who would have stayed, which is 9.5 points, so the line is at 23.8%, a long way from 14.3%. If the price changes nothing, showing that the new-price group stays below the line takes 213 customers in each group.
But 213 assumes the rise costs nothing. A rise that does cost some customers lands closer to the line, and the closer it lands, the more customers it takes to show it is below. If the rise costs half the line, the test needs 959 a group, which still fits Starter’s 2,400. E9 makes the same point about trials whose size assumes no difference. The example decision memo runs this test and reads it out.
A lasting rise in churn: more customers than the plan has
The first-wave line treats the higher price as a one-off shock. If it instead keeps customers leaving faster every month, the line is far tighter. At a steady churn rate, the average customer stays 1 ÷ churn months: 20 months at 5% a month, worth 20 × €23.20 = €464 of margin, undiscounted. At €31.90 the same €464 comes in 17.8 months, so the rise pays only while monthly churn stays below 5.63%, 0.6 of a point above today.
Over a year, that is 46.0% of customers leaving against 50.1%. If the price changes nothing, showing that the year’s leaving stays below 50.1% takes 2,303 customers a group, 4,606 in all: almost twice the plan’s customers, and a year of waiting. With half its customers at each price for a full year, it could reliably see monthly churn rising to about 5.9%, but not to 5.63%.
Why small effects take so many customers
The gap sits squared at the bottom of the formula, so the customers needed grow with the square of how fine a difference you want to see. Halve the gap and you need four times as many; double it and you need a quarter: against a line twice as far away, the sign-up test would need 954 a group instead of 3,815. A break-even stated as a share of a rate is a small gap: 2.8 points on a 25% conversion rate, 0.6 of a point on 5% monthly churn. The first-wave line is wide only because it is a share of the 85.7% of customers who would have stayed.
Twice the gap, a quarter of the customers: 954 a group instead of 3,815
The same data as a table
| Gap, points | Customers a group |
|---|---|
| 2.0 | 7,359 |
| 2.8 (the break-even) | 3,815 |
| 4.0 | 1,840 |
| 5.6 (twice the break-even) | 954 |
| 6.0 | 818 |
| 8.0 | 460 |
| 10.0 | 295 |
The usual mistake follows from this. A test too small to see the line finds “no significant difference”, and the price rise is declared safe (in a board meeting, the two sound alike). It has shown nothing. E9 calls it inappropriate to conclude that a new drug is no worse from a test that merely failed to find a difference, and a price test is no different. A test shows a rise is safe only when the whole range of its likely effect sits below the break-even.
Only a range wholly below the break-even shows a price rise is safe
What to do when the numbers are out of reach
Test where the line is far away. That is usually existing customers’ first reaction, measured in waves with a stop rule agreed before anyone sees the new price, as the pricing analysis page sets out.
Accept less certainty where the decision is cheap to reverse. At 80% confidence instead of 95%, the formula needs 43% fewer customers. A price shown only to new sign-ups can be changed back next month, which makes it a fair case for E9’s other values. Here it shortens the sign-up test from 13 months to 8, inside a year.
Decide the rest on other evidence, and say so. Keeping both groups on their prices for a year can catch a large lasting rise in churn, though not one at the break-even. Conversion can be watched against the same months a year earlier, which is weaker evidence than a test, because a campaign or a change in where sign-ups come from moves it too. The decision memo should say which of its numbers came from a test and which did not.
My read: at 600 sign-ups a month, this company can test its existing customers’ first reaction in one wave, and its sign-ups in about a year, or 8 months at 80% confidence. A lasting rise in churn at the break-even it cannot test at all. That part is judgement, and it is better made knowing that it is judgement.
How FinDaS does it
A FinDaS price test starts with the break-even and the group sizes, agreed before anyone sees a new price, and its design memo says which questions the company’s numbers can answer and which it is deciding on judgement. AI agents build the analysis, automated checks and a review round check every deliverable, and each carries a short note of which checks ran. What you act on, such as a model going live or a decision memo, goes to a panel of reviewer agents, and then I sign it; routine refreshes do not carry my signature. The method, and what it costs, is on pricing analysis.
If an AI tool has already given you a test result, there are six checks to run on its numbers in How to check the numbers in an AI-generated report.
Sources
- International Conference on Harmonisation, E9: Statistical Principles for Clinical Trials, Step 4 version, 5 February 1998. Sections 3.3.2 and 3.5.