A/B Testing Exercises in R: 20 Real-World Practice Problems

Explanation: Aggregating by variant with n(), sum(), and mean() gives the three numbers every A/B report starts with: sample size, conversion count, and conversion rate. The mean() shortcut works because converted is coded 0/1, so the average equals the proportion. Always print user counts alongside rates so a stakeholder can sanity-check sample sizes before reading the rate.

Explanation: prop.test() is the default tool for binary outcomes: it pools the proportions under the null and computes a chi-square statistic equivalent to a two-sided z-test. Setting correct = FALSE matches the textbook z-test; the default Yates continuity correction is conservative and rarely matters for the large samples typical of online experiments. With p-value 4e-06 and a CI that excludes zero, the treatment is statistically detectable.

Explanation: A 2-by-2 chi-square on counts is algebraically identical to the two-proportion z-test in Exercise 1.2: same statistic (21.21), same p-value, same conclusion. Use the matrix form when you already have counts in a cross-tab (e.g., from xtabs() or count() |> pivot_wider()); use prop.test() when you have raw numerators and denominators. The trap to avoid: forgetting correct = FALSE if you want to match the z-test exactly.

Explanation: The one-sided p-value is exactly half the two-sided p-value when the direction matches the data, so 4.1e-06 becomes 2.1e-06. The PM can confidently say the treatment is higher (p well below 0.05). Two cautions: the directional choice must be pre-registered before peeking, otherwise running both directions and picking the smaller p-value silently doubles your false-positive rate. Also note that prop.test() takes the variants in the same order you pass x and n, so swap them carefully when asking for greater.

Explanation: t.test() defaults to Welch's t-test, which does NOT assume equal variances and is the right choice for nearly every A/B test on revenue or session metrics. The CI for the mean difference (control minus treatment) excludes zero by a hair and the p-value is just under 0.05. With small samples (n=10 per arm) the interval is wide, so even a "significant" result like this is fragile: that one-sample-flip from significance is exactly why you size experiments before peeking, which is the next section.

Explanation: With nearly identical sample sizes and similar variances, Welch and pooled produce almost the same answer: df 18.0 vs 17.95, p-values rounding to 0.05. Welch only differs meaningfully when variances are unequal AND group sizes are unequal. The cost of using Welch when variances ARE equal is essentially zero (slightly less power, never wrong), but the cost of pooling when variances are unequal is an inflated false-positive rate. Default to Welch unless you have a strong reason.

Explanation: Raw revenue distributions almost always have a long right tail that violates the normality assumption of the t-test and inflates variance. Taking log1p() (which is log(1 + x) and handles zero values safely) symmetrizes the distribution and the t-test on log-scale means becomes a test on geometric means, often the more meaningful summary for revenue. A common alternative is the Mann-Whitney U test via wilcox.test(), but the log-t approach is preferred when you care about quantifying the effect size as a multiplicative lift.

Explanation: The output n is per group, so the total user count is twice that. A useful rule of thumb falls out: smaller minimum detectable effects (MDE) need quadratically more users, so halving the MDE from 1pp to 0.5pp would require roughly four times the sample. Always pre-compute this BEFORE launching, not after a marketing campaign drives unplanned traffic. Leave any one of n, p2, or power as NULL to solve for it; the function fills in the missing slot.

Explanation: pwr.2p.test() parameterizes the effect by Cohen's h (an arcsine-transformed difference) rather than two raw proportions. For h = 0.05 the function returns about 3,140 per group, much smaller than Exercise 3.1's 14,744 because h = 0.05 corresponds to a larger relative effect than the 10pp-to-11pp jump (Cohen's h for that comparison is only about 0.033). The arcsine transform stabilizes variance across the proportion scale, which is why pwr uses it. Convert between formulations with pwr::ES.h(p1, p2).

Explanation: Solving for p2 with fixed n flips the usual workflow: instead of "how many users do I need?", you ask "what is the smallest effect I can plausibly detect with what I have?" The honest answer for 8,000 per arm is a 13.6% relative lift, which lets the PM decide whether the test is worth running. If the realistic business effect is a 3% relative lift, this experiment is underpowered and should be redesigned (longer runtime, more variants ruled out, or a larger primary metric).

Explanation: This is the classic Wald interval for two proportions and matches the CI that prop.test() reports (with sign flipped depending on which proportion you subtract). The interval lies entirely above zero so the lift is statistically detectable. For very small or very large proportions (p < 0.05 or p > 0.95), the Wald approximation is poor and the Wilson interval (set correct = FALSE and use binom.test() or prop.test()) is the better default. Reporting the CI alongside the p-value is far more informative for stakeholders than the p-value alone.

Explanation: A bootstrap percentile CI makes no normality assumption, which matters because revenue is heavily skewed and the t-test's symmetric CI on the raw scale would be misleading. The CI here spans negative to positive, so the experiment cannot rule out either a loss or a sizeable win: under-powered. Two practical notes: use replicate(R, ...) or vectorize with matrix sampling for speed on large data, and prefer BCa CIs (boot::boot.ci(type = "bca")) over plain percentiles when the bootstrap distribution is skewed or biased.

Explanation: Cohen's h transforms two proportions onto an arcsine scale where the SD is approximately constant, then takes the difference: h = 2 * (asin(sqrt(p1)) - asin(sqrt(p2))). Conventional thresholds: 0.2 small, 0.5 medium, 0.8 large. A value of 0.033 is tiny, which is why Exercise 3.1 needed nearly 30,000 total users: the smaller the effect on the arcsine scale, the more samples you need. Use ES.h() whenever you need a size-free way to compare experiments with different baselines.

Explanation: With 4 variants there are 6 pairwise tests, so Bonferroni multiplies each raw p-value by 6 and caps at 1.0. Only C vs D survives at adjusted p = 0.02. Reporting the unadjusted p-values from 6 separate prop.test() calls would inflate the family-wise error rate well above 5%. Pick the comparison method to match your goal: Bonferroni for strong control of family-wise error, BH (the next exercise) for control of false discovery rate when you have many comparisons and are tolerant of a few false positives.

Explanation: BH controls the expected proportion of false discoveries among rejections, which is usually what you want when running many parallel tests: you tolerate a few false positives in exchange for higher power than Bonferroni. Bonferroni would shrink raw 0.020 to 0.20 (rejecting nothing past the first two), while BH keeps three discoveries. Use BH for screening (which features are worth deeper analysis?), Bonferroni for confirmatory comparisons where any false positive is expensive (regulatory submission, public claims).

Explanation: Holm (step-down) is uniformly at least as powerful as Bonferroni while controlling the same family-wise error rate, so there is no reason to prefer Bonferroni over Holm for confirmatory comparisons. Bonferroni multiplies every p-value by m (the number of tests); Holm sorts p-values and uses the multiplier m - rank + 1, which is smaller for all but the smallest p-value. Default to method = "holm" for FWER control and method = "BH" for FDR control: that pair handles 95% of A/B testing needs.

Explanation: Even when both variants have identical true rates, early days show wide gaps that close as sample size grows. This is exactly the trap PMs fall into when peeking: they see a gap on day 3, conclude treatment is winning, and stop. The fix is either to commit to a fixed-horizon analysis (run for the pre-computed sample size, then look once) or to use a sequential procedure that adjusts for repeated looks (alpha spending, Bayesian bandits). The chart is a great visual aid for explaining why peeking is dangerous.

Explanation: A naive 7-look procedure has effective alpha far above 0.05 (the actual inflation is roughly 0.30 when looks are independent), so any "significant" finding mid-experiment is mostly noise. Bonferroni-adjusting to per-look alpha 0.0071 controls the family-wise error at 0.05 conservatively. Better procedures (Pocock, O'Brien-Fleming, mSPRT) spend alpha unevenly across looks for higher overall power, but Bonferroni is the right starting point if you have to invent a rule under time pressure.

Explanation: Sample Ratio Mismatch is the most common operational bug in production experimentation: a 50/50 randomizer that bucketed at 48.4/51.6 is wildly off and the p-value confirms it isn't sampling noise. Real causes include bot filtering that strips one arm asymmetrically, opt-in flows that gate the treatment, or assignment code that runs after a pre-treatment redirect. Any A/B test result with SRM detected is invalid: do not patch the analysis, fix the root cause and rerun. A common dashboard threshold is p < 0.001.

Explanation: A correctly calibrated test rejects the null at the nominal alpha rate when the null is TRUE. Here the empirical false-positive rate is 0.046, within Monte Carlo error of the theoretical 0.05. Running an A/A simulation on your real production pipeline (using actual traffic split, not just rbinom) is the highest-value sanity check before any A/B program launches: if you observe inflated false positives, your test is either using the wrong statistical formula or your randomizer is biased. Bookmark this as a regression test.