Subscription A/B Test Sample Size Calculator

01 · Two-proportion sample size

n = [ z_(1-α/2)·√(2·p̄·(1-p̄)) + z_(1-β)·√(p1·(1-p1) + p2·(1-p2)) ]²
    ─────────────────────────────────────────────────────────────
                          (p2 - p1)²

where p̄ = (p1 + p2) / 2

The pooled variance (2p̄(1-p̄)) controls the false-positive term because that's the true variance of the test statistic under the null hypothesis (p1=p2); the unpooled variance (p1(1-p1)+p2(1-p2)) controls the power term because that's the true variance under the alternative. Two-sided, normal approximation, no continuity correction.

Source: Dalgaard, P. (2008). Introductory Statistics with R, 2nd ed., Springer, §9.1.4. This is the same derivation as Fleiss, Levin & Paik, Statistical Methods for Rates and Proportions, 3rd ed., Wiley, 2003, and as R's stats::power.prop.test.

02 · Two-stage funnel dilution

p_overall = p(trial_start) × p(trial_to_paid)

If the experiment changes only one stage, the sample size is computed on the resulting overall rate (formula 01, applied to p_overall), not on the targeted stage's own rate. A relative lift on the targeted stage passes through to the overall rate unchanged, but the absolute effect, the thing the sample-size formula actually depends on, shrinks by the downstream stage's conversion rate. This is the single most common reason a mobile paywall test is underpowered without anyone noticing.

Direct consequence of the definition of a two-stage funnel (P(A and B) = P(A)·P(B|A)). Not a citation-requiring result on its own.

03 · Revenue-per-install (continuous metric) sample size

n = 2 · σ² · (z_(1-α/2) + z_(1-β))²
    ────────────────────────────
              Δ²

σ = CV × mean,   Δ = MDE × mean

Standard two-sample, two-sided, equal-variance normal approximation. Because both σ and Δ scale with the mean here, the mean cancels out of n algebraically when the MDE is relative. The coefficient of variation, not the absolute revenue number, is what drives sample size. Subscription revenue per install is heavy-tailed (a handful of annual or lifetime purchases can dominate the variance), and this normal approximation degrades badly at the sample sizes indie teams actually run. Treat this number as a floor, not a guarantee.

Source: standard two-independent-means result, e.g. Dalgaard (2008) §9.1.3.

04 · Test duration

days = ceil(n_per_variant × n_variants / daily_views) + trial_days + settlement_days

The last cohort enrolled is the one that gates when the test can honestly be read: a 7-day trial plus a settlement buffer for billing retries means that cohort isn't measurable for well over a week after it enrolls, not from the day the test started.

On peeking

The result above is a fixed-horizon sample size: it assumes the test is read once, at the end. Monitoring continuously and stopping at the first apparent win inflates a nominal 5% false-positive rate to roughly 26% (checking every ~500 visitors). This tool deliberately does not offer a sequential/alpha-spending correction (e.g. O'Brien-Fleming boundaries) alongside the fixed-horizon number. That is a materially harder problem to implement correctly, and an unverified version of it would be worse than not shipping one at all.

Source: Pekelis, L., "Statistics for the Internet Age: The Story Behind Optimizely's New Stats Engine", Optimizely, 2015; formalized in Johari, Koomen, Pekelis & Walsh, "Peeking at A/B Tests: Why it Matters and What to Do About It", KDD 2017.