A/B Testing Interview Cases: 20 Solved in R

Explanation: power.prop.test() inverts the two-proportion power formula: give it any four of {n, p1, p2, power, sig.level} and it solves for the fifth. Here n is left out, so it returns the per-arm size (about 7760 users) which you round up. A common interview trap is forgetting this is per arm, so total enrolment is roughly double. Smaller baseline rates or smaller lifts both inflate the count quickly.

Explanation: The minimum detectable effect (MDE) is the smallest true difference an experiment is powered to catch. With 6000 per arm the detectable treatment rate is about 13.71%, an MDE near 1.71 absolute points, which is larger than the 1.5-point lift the team cared about in 1.1. That gap is the honest interview answer: this traffic budget is underpowered for the effect they hope to see, so a null result would be uninformative.

Explanation: Power rises with sample size but with diminishing returns. Going from 5000 to 10000 users lifts power from 0.61 to 0.89, while the next 10000 only buys 0.10 more. Interviewers use this to check whether you understand that doubling traffic does not double your ability to detect an effect, and that chasing the last few percent of power can cost far more traffic than the earlier gains bought.

Explanation: Pooling the totals gives a traffic-weighted baseline (about 11.93%), which differs from the simple mean of the seven daily rates whenever daily traffic is uneven, as it is here with lighter weekends. Using the weighted figure matters because the sample size math in 1.1 assumes this population rate. Averaging the daily percentages instead would over-weight low-traffic days and bias the baseline downward or upward unpredictably.

Explanation: The two-proportion test asks whether the gap between 13% and 12% is larger than sampling noise. Here X-squared is 9.14 with a p-value of 0.0025, so we reject the null of equal conversion. Turning off the continuity correction with correct = FALSE reproduces the classic z-test, since a chi-square on one degree of freedom is the square of that z. With the correction on, the p-value is slightly larger and more conservative.

Explanation: The interval runs from 0.0035 to 0.0165, meaning the true lift sits between about a third of a point and 1.65 points with 95% confidence. Because it lies entirely above zero, the result agrees with the significant p-value from 2.1. Interviewers like the confidence interval because it communicates effect size and precision together, whereas a p-value alone hides whether a win is trivially small or genuinely business-moving.

Explanation: The absolute lift is one point, but relative to a 12% base that is an 8.33% improvement, the figure most stakeholders and dashboards report. Confusing the two is a classic mistake: a "50% lift" could mean a move from 2% to 3% (huge relatively, tiny absolutely). Always state which one you mean, and remember that the sample size math is driven by the absolute gap, not the relative one.

Explanation: The one-sided p-value (0.00125) is almost exactly half the two-sided value (0.0025) from 2.1, because the test now spends all of alpha in the upper tail. One-sided tests are more powerful but only honest when the direction is fixed in advance for a real reason. Deciding to go one-sided after seeing which way the data points is a form of p-hacking that interviewers will probe for.

Explanation: With no true effect, a single test rejects near the nominal 5% (here 0.059), exactly as designed. But allowing yourself to stop at any of ten looks pushes the false positive rate to about 21%: roughly one A/A test in five would be declared a false winner. This is the peeking problem, and it is why calling an experiment the moment it crosses significance is one of the most damaging habits interviewers screen for.

Explanation: Under the independence assumption, ten looks push the false positive rate from the nominal 5% to about 40%. Real sequential data is positively correlated across looks, so the true inflation from 3.1 (near 21%) is lower than this upper bound, but the direction is identical: naive repeated testing guarantees far more false wins than the stated rate. This is exactly why teams adopt alpha-spending or group-sequential boundaries instead of testing continuously.

Explanation: Bonferroni divides the 5% budget across the five looks, so each must beat 0.01. Only look 4 (p = 0.006) clears it, even though three looks were below the naive 0.05. Bonferroni is simple and always valid but conservative, because it ignores the correlation between looks on accumulating data. Group-sequential methods like Pocock or O'Brien-Fleming spend alpha more efficiently, which is the follow-up interviewers often want to hear next.

Explanation: The binomial standard error $\sqrt{p(1-p)/n}$ treats every session as independent, which is fine when the randomization unit really is the session. The catch, and the reason this is only a naive SE, is that sessions cluster within users: one very active user contributes many correlated sessions, so the effective sample size is smaller than 8200 and the true SE is larger. The next exercise handles a ratio where that structure bites.

Explanation: The delta method linearizes the ratio around the mean number of sessions, giving $\operatorname{Var}(R) \approx \frac{1}{n\,\bar{Y}^2}\left(\operatorname{Var}(X) - 2R\operatorname{Cov}(X,Y) + R^2\operatorname{Var}(Y)\right)$. It captures how variation in the numerator, the denominator, and their covariance all feed the metric's uncertainty. Treating this ratio as a simple proportion would misstate the SE. Getting the delta-method formula right for a ratio metric is a classic senior-level screening question.

Explanation: The bootstrap SE (0.00350) lands almost exactly on the delta-method value (0.00354), which is the reassurance you want: two independent routes to the ratio's uncertainty agree. Resampling users rather than sessions is the crucial detail, since it respects the randomization unit and reproduces the clustering. When a closed-form delta method is awkward, this user-level bootstrap is the general-purpose fallback for any ratio or funnel metric.

Explanation: CUPED (Controlled experiment Using Pre-Experiment Data) removes the part of the outcome that the covariate already explains, shrinking variance by roughly the squared correlation between them (here about 35%). Because the covariate is centered and predates assignment, it cannot be affected by treatment, so the adjusted metric stays unbiased while its confidence interval tightens. Less variance means the same power at a smaller sample size, which is why CUPED is standard at large experimentation platforms.

Explanation: Week 1 averages a 2.64% lift while week 2 collapses to 0.15%, the signature of novelty: users react to the newness, not the lasting value. Shipping on the week-1 number would badly overstate the long-run impact. The interview point is that a single pooled estimate hides these time dynamics, so plotting or grouping the effect over time is essential before trusting a positive readout, especially for prominent UI changes.

Explanation: The two weekend days convert near 11.3% while weekdays sit around 13.5%, a large day-of-week pattern. If the treatment and control arms were not balanced across weekdays (for example if the test started mid-week), pooling would confound the treatment effect with this weekly seasonality. Blocking or covariate-adjusting for weekday, or simply running whole weeks, removes the confound. Interviewers use this to test whether you check assignment balance before believing a result.

Explanation: A good readout answers two questions at once: how big (0.90 absolute points, 7.5% relative) and how sure (p = 0.0028, significant). Reporting only the relative lift flatters small effects, while reporting only the p-value hides the size. Pairing them, ideally with the confidence interval from 2.2, is what separates a decision-ready summary from a number dump. This is the exact table an interviewer expects you to build from raw counts.

Explanation: With a p-value near 3e-08, the 24000 versus 22800 split is wildly inconsistent with a fair coin, a textbook sample ratio mismatch (SRM). Teams flag an SRM whenever this p-value drops below about 0.001, because it signals a broken assignment, a logging bug, or bot filtering that differs by arm. When an SRM fires, the correct move is to debug the pipeline and discard the readout, not to interpret the lift: no significance test downstream is trustworthy.

Explanation: The p-value is 0.24, so the observed 0.4-point lift is not significant, but that does not prove treatment did nothing. With 10000 per arm the test was only powered to reliably detect a lift of about 0.98 points (to a 6.98% rate), more than double the effect seen. The honest conclusion is underpowered and inconclusive, not "no effect." Confusing absence of evidence with evidence of absence is one of the most common interview traps.