Did the promotion pay?

How it works

The test. An online coffee shop with 45,000 customers who ordered in the last year sells 20 products in five categories, from paper filters at €5 to an espresso machine at €349: about 12,500 orders a month, at an average order of €40.54. On Monday 3 August 2026 it emailed a code for 15% off every order for two weeks to half of those customers, picked at random; the other half got nothing. Both halves were watched for four weeks before the code and eight weeks after it ended, to 11 October 2026. Every order the half with the code placed in those two weeks used it.

Random halves. A seeded shuffle, not a person, picked who got the code, so the halves differ only by chance. The half without the code shows what the other half would have bought without it, week by week, the shop’s own busy and quiet weeks included. In the four weeks before the code, the half that got it placed 5,834 orders and the other half 5,863: a difference of −29, inside its 95% interval of −266 to +208.

Difference in means. For every customer, the readout adds up the weeks counted: orders, revenue at full price, the discount used, and the margin after discount, which is the gross margin times revenue at full price, less the discount. The code’s effect is the average customer in the half that got it less the average customer in the half that did not, times the 22,500 customers who got it. At the defaults, margin after discount averages €9.43 a customer in the half with the code and €10.40 in the half without: −€21,800 over the half.

The interval. Customers vary, so the difference carries noise. Its standard error is √(s₁² / n₁ + s₂² / n₂), from each half’s standard deviation and its customers: √(22.94² / 22,500 + 25.46² / 22,500) = €0.228 a customer, or €5,140 for the half. The 95% interval is the difference plus or minus 1.96 standard errors: −€21,800 ± €10,100, from −€31,900 to −€11,700.

Pull-forward. An order bought in the code’s weeks at 15% off is sometimes an order the customer would have placed a few weeks later at full price. It shows up as one fewer order in a week after, which is why the shop watched eight more weeks. Counted over the code’s two weeks only, the code looks paid, +€7,700 (95% interval +€3,500 to +€11,800); counted over all eight weeks after, it lost €21,800.

The discount on orders that would have come anyway. The discount given is 15% of the revenue at full price of every order that used the code, €32,500. The part on the extra orders is 15% of the extra revenue, €4,000; the other €28,500 went to customers on orders they would have paid full price for.

The verdict. Paid, when the whole 95% interval of the margin left is above zero. Did not pay, when the whole interval is below zero. Cannot tell yet, when it spans zero: at that margin this test is too small to tell a loss from a gain, and a bigger one settles it.

What it does not cover. One depth of discount: what 10% or 25% off would have done is not in this test’s data, and only a half sent that code measures it, so depth is not one of the controls. The code went to every customer who ordered in the last year; a code for lapsed customers alone is a different test. Customers who joined after the split are in neither half, orders more than eight weeks after the code are not counted, and the cost of sending the email is left out.

The same shop elsewhere. The market basket demo finds which of this shop’s products sell together, and the loyalty demo asks whether its points programme pays for itself.