A/B Testing Exercises in R: 18 Real-World Practice Problems

Explanation: ES.h() converts two proportions into Cohen's h, an arcsine-transformed effect size that stabilises variance across the 0-1 range. Plugging h into pwr.2p.test() lets you solve for any one missing piece (n, power, sig.level, or h); pass three and leave the fourth as NULL. A common mistake is plugging the raw difference p1 - p2 = 0.01 instead of h: that conflates effect size with proportion units and undersizes the test by roughly 20% at low baselines.

Explanation: Cohen's d for two samples is (mu1 - mu2) / sd_pooled; here it collapses to 4 / 32 = 0.125, a "small" effect. The type = "two.sample" argument is critical: dropping it defaults to a one-sample test, which dramatically undersizes the experiment. For unequal sds use pwr.t2n.test() with the more conservative pooled sd, or simulate power directly since pwr.t.test() assumes equal variances under the hood.

Explanation: With a sample cap, the meaningful question flips from "how many users?" to "how big a lift must we believe in?". The h returned by pwr.2p.test() is in arcsine units; inverting 2*asin(sqrt(p)) back to a proportion gives the detectable treatment rate. At 4% baseline with 5,000 per arm, you can only see lifts of ~29% relative or larger; smaller lifts will look like noise. This is the right diagnostic to run before launching, not after a flat result.

Explanation: Power curves communicate experiment cost far more effectively than a single "need 14,000 users" number. The curve is concave: doubling n from 2,000 to 4,000 buys you 22 power points; doubling again from 10,000 to 20,000 only buys 12. Plot this with geom_line() and add a horizontal line at 0.8 so stakeholders can read off the inflection point. sapply() over the n grid is fine here; for many parameter combinations, purrr::map_dfr() over an expand_grid() is cleaner.

Explanation: prop.test() is the workhorse two-sample comparison: pass conversions as x and totals as n, both length-2. By default it applies Yates' continuity correction, which inflates the chi-square statistic slightly and is conservative at small counts; pass correct = FALSE for the uncorrected z-test that most modern A/B platforms report. The CI here is for prop 1 - prop 2, so a wholly negative interval means treatment beats control; flip your sign convention only if your stakeholder reports lift as treatment - control.

Explanation: Wrapping prop.test() in broom::tidy() is what turns a printable htest into a row you can bind across dozens of tests. Use this inside purrr::map_dfr() when you sweep over many metric/segment combinations; you get a single tibble where each row is one A/B comparison, ready for arrange(p.value) or mutate(p_adj = p.adjust(p.value, "BH")). glance() is an alternative for some htest classes but tidy() is the right choice for prop.test because it surfaces both estimates and the CI in one row.

Explanation: For a 2x2 table chisq.test() and prop.test() produce identical p-values; both reduce to the same chi-square statistic on one degree of freedom. The matrix form is more natural when you have churn pulled from a SQL GROUP BY variant, churned query. Use chisq.test(m)$expected to inspect expected counts; if any cell drops below 5 (rare with experiment-scale data but common in stratified slices) reach for fisher.test() instead.

Explanation: Welch is the right default for revenue-style metrics because variance often differs between arms; t.test() uses Welch unless you pass var.equal = TRUE. Note the true mean difference is $2 but the observed sample difference (~$2) is not significant at n=1,000 because $32-$33 sds make the standard error large (sqrt(32^2/1000 + 33^2/1000) ~= 1.45). Exercise 1.2 showed you needed ~1,004 per arm just to detect a $4 lift; here we asked for half that effect with the same n, so a null result is expected.

Explanation: wilcox.test() (Mann-Whitney U) compares stochastic dominance instead of means, so it is robust to the long right tail typical of latency, session duration, and revenue distributions. The null is "P(X > Y) = 0.5"; rejecting it means one distribution tends to produce larger values, not that means differ. For very large samples, prefer permutation tests or rank-based bootstrap CIs over the asymptotic Wilcoxon, but for n=2,000 the continuity-corrected version is fine.

Explanation: Simulation is the most reliable power tool when the data-generating process violates t-test assumptions. Here the true means differ by 1.0 (10 vs 11) and the analytic formula would suggest ~50% power, but the skewed exponential distribution inflates within-group sd so realised power is closer to 21%. The fix is either log-transform revenue before testing, use Wilcoxon, or apply CUPED variance reduction with a pre-period covariate. Always benchmark your test plan with a quick simulation before committing engineering time.

Explanation: Even though every individual peek is a valid 5%-alpha test, the union across 10 looks gives roughly 4x the nominal false positive rate (~21% vs 5%). This is the central problem with watching dashboards in real time. The right fixes are alpha-spending procedures (O'Brien-Fleming, Pocock), Bayesian sequential tests with proper priors, or simply running to the pre-registered fixed horizon. Never stop a test early just because "it crossed the line today".

Explanation: Bonferroni divides alpha evenly across all planned tests: 0.05/5 = 0.01 per look. It is conservative because the looks are correlated (each later look reuses earlier users), so the realised FPR (~5%) lands close to the target. Alpha-spending functions like O'Brien-Fleming spend alpha non-uniformly (almost nothing early, most at the end), achieving better power than Bonferroni for the same FPR ceiling, but Bonferroni is the easiest to explain in a stakeholder doc.

Explanation: Mid-experiment power recalculation is fine as a planning exercise; the trap is conditioning on the observed effect and then declaring "we will continue until significance". That conditioning bias inflates FPR. The right framing for a stakeholder is: "the observed effect is consistent with both the null and a 12% relative lift; to be 80% sure we could detect a 12% lift we would need ~12,400 per arm; given current traffic that means ~4 more weeks. Should we commit?". A decision, not a guarantee.

Explanation: Dropout inflation is n_design / (1 - dropout); never multiply by (1 + dropout) since that under-inflates. The deeper issue is whether dropout is random or related to the treatment itself (differential attrition), which is a far more serious threat to validity than just "we have fewer rows". Always report attrition by arm in the analysis section; a 5pp gap between arms should trigger a sensitivity analysis with inverse-probability weighting before declaring a winner.

Explanation: Bonferroni is the strictest correction (multiply each p by m=6), controlling family-wise error rate, while Benjamini-Hochberg controls the expected proportion of false discoveries among rejections. With 6 metrics here, Bonferroni keeps only m1 significant at 0.05, while BH keeps m1 through m4. Use Bonferroni when a single false positive would be costly (e.g. drug approval); use BH when you are screening many candidate metrics and can tolerate a small false-discovery proportion. Avoid the temptation to skip correction entirely; uncorrected secondary metrics are how teams ship features that look like they help across "some metric".

Explanation: A negative slope on week_num is the signature of novelty: users react to the change initially, then revert. The point estimate of -0.0047 per week means roughly 0.5 percentage point of lift evaporates every week. The right stakeholder framing is "do not size launch decisions on week-1 results"; the test should run at least 2 to 4 weeks to let the trend stabilise, then estimate steady-state lift from the last week. For a more rigorous version, use a segmented regression or piecewise model to identify the changepoint.

Explanation: prop.test() returns the CI for p1 - p2 (control minus treatment); since stakeholders prefer "lift = treatment minus control", flip the sign and the order of the interval bounds. Reporting both absolute lift in percentage points AND relative lift in percent is non-optional: "+0.3 pp" reads small, "+7.4% relative" reads big, and both are true. The decision column codifies the launch rule so engineering does not relitigate it after the fact; for more nuance add tiers like "Ship", "Hold", "Iterate", or "Investigate" based on directional CI and effect size.

Explanation: An A/A test catches two kinds of bugs: a broken randomiser (traffic skew) and a broken event pipeline (conversion delta despite identical treatment). Always run one BEFORE shipping a real experiment, especially after changes to the bucketing layer or analytics SDK. If the traffic split test fails, fix the randomiser before trusting any A/B result; if only the conversion test fails, suspect a logging bug like deduplication misfiring on one variant.