Experiment design
Will a price rise pay, and can a test prove it first?
A break-even and power calculation: how many customers a price rise can lose and still pay, and how large a test must be to prove it before rollout.
The answer for a 10% rise on the Starter plan
Break-even
11.1%
A 10% rise at a margin of 80% earns more as long as fewer than 11.1% of paying customers leave because of it: up to 266 of Starter's 2,400.
Can a test prove it first?
Yes
On existing customers: 213 a group shows the extra leaving stays under the break-even, so one wave of Starter's 2,400, in two groups of 1,200, read after three months.
The same test on sign-ups
13 months
Sign-ups need 3,815 a group, at 300 a group a month: past the 12-month limit.
Size your own test: a power calculation
On today's price.
What is left of the price after the cost of serving a customer.
The share of trials that convert, or of customers who leave, over the period the test counts.
A unit example: from 25% to 23% is 2 points.
Sign-ups a month, halved. For existing customers, how many go into each group at once.
Break-even line
9.5 pts
A rise of 10% at a margin of 80% pays while the extra leaving stays under 11.1% of the 85.7% who would have stayed.
Can a test prove it within a year?
Yes
213 customers a group, if the price changes nothing: one wave of 1,200 a group holds them.
Customers a group to see the smallest change
5,082
To see a 2.0-point rise in leaving from 14.3%, at 95% confidence and in 8 tests of 10: more than the 1,200 a group the customer base holds.
| True change from the price | Points | Customers a group | To fill each group |
|---|---|---|---|
| None | 0.0 | 213 | One wave |
| A quarter of the line | 2.4 | 403 | One wave |
| Half the line | 4.8 | 959 | One wave |
| Three quarters of the line | 7.1 | 4,031 | More than the base |
Why sign-ups cannot test the rise, and existing customers can
At 600 sign-ups a month, 300 go into each group. Seeing a 2.0-point fall in conversion, from 25.0% to 23.0%, takes 7,155 a group: 24 months. The break-even line sits further out. A rise of 10% at a margin of 80% pays while conversion falls by less than 11.1% of itself, to no lower than 22.2%: a line 2.8 points away. Showing that the fall stays inside it takes 3,815 a group if the price changes nothing, 13 months. Doing it within 12 months would take 318 a group a month.
Existing customers are a different test. Starter’s 2,400 customers go into two groups of 1,200 at once, and over three months 14.3% of a group leaves at today’s price. The line is 11.1% of the 85.7% who would have stayed: 9.5 points of the group, more than 3 times as far out as on sign-ups. Showing that the extra leaving stays inside it takes 213 a group, so one wave of 1,200 is enough. Measuring a 2.0-point rise in leaving would take 5,082 a group, more customers than Starter has.
The difference is what the line is a share of. On sign-ups, it is a share of the 25.0% who convert; on existing customers, of the 85.7% who stay. The example decision memo runs the Starter test and reads it out.
The table: where the true change sits
A price rarely changes nothing. The closer the true change sits to the line, the more customers the test needs to show it is still inside: on Starter, 213 a group at no change, 959 at half the line and 4,031 at three quarters. Read the row for the change you expect.
How the numbers are worked out
The break-even. A rise of r on a margin m, both as shares of today’s price, pays until r ÷ (m + r) of the paying customers leave because of it: 10% ÷ (80% + 10%) = 11.1%. A customer who would not pay at today’s price either does not count, so in a test the line is that share of the customers who would have stayed, or of the trials that would have converted.
Group size, the power calculation. Each group needs (1.96 + 0.84)² × (p1(1 − p1) + p2(1 − p2)) ÷ gap² customers, rounded up. p1 and p2 are the two groups’ true rates; 1.96 sets 95% confidence, and 0.84 finds a real change in 8 tests of 10. To see a change, the gap is the change. To test the line, it is the distance from the true change to the line.
Months, or one wave. On sign-ups, the group size divided by the customers who enter each group a month, rounded up; the read-out comes once the last of them has been through the period the test counts. Existing customers are a fixed base: they go into the groups at once, in one wave, so the question is whether the base holds the groups, not how many months it takes.
Within 12 months. A test that takes longer counts as not testable: over a longer test, other changes to the product and the market pile up on top of the price.
Where the defaults come from
The company is the one in the revenue forecast demo: 600 sign-ups a month, 25.0% of trials converting to its Starter plan at €29, and 2,400 Starter customers, 5.0% of whom leave a month. The margin, 80%, the rise, 10%, and the smallest change worth seeing, 2 points, are set for this calculator. None of it is a benchmark or a client’s figure.
Not covered
The calculator sizes a test that compares two shares: trials that convert, or customers who leave. It does not size a test of revenue per customer or of moves to a cheaper plan, and it does not set the price. It assumes the groups are chosen at random and read once, at the end. A test read every week and stopped when it looks good needs larger groups than these. How a price test fits into a pricing decision is on pricing analysis.
Start here
Want a price test sized on your own customers?
Up to thirty minutes on the change and your data, then a fixed proposal.