Multiple Testing Exercises in R: 18 Solved p.adjust() Problems

Explanation: Bonferroni is the simplest correction in the family: multiply every raw p-value by the number of tests m, then cap at 1. The cap matters because 0.50 * 5 = 2.5 is not a valid probability. pmin() does the elementwise floor against 1. Bonferroni controls FWER (the probability of any false positive) at the chosen alpha, but pays for that guarantee with low power once m grows: with 1000 tests, a raw p of 0.0001 becomes 0.10 and stops looking interesting.

Explanation: Holm is a step-down version of Bonferroni: the smallest raw p is compared to alpha/m, the next to alpha/(m-1), and so on. The adjusted form rescales each by its rank-dependent multiplier. The cummax step is the crucial bit: once you fail to reject at rank k, every larger rank is automatically not-rejected, which is enforced by making the adjusted vector non-decreasing. Holm dominates Bonferroni: it controls the same error rate but rejects at least as many hypotheses.

Explanation: Benjamini-Hochberg targets a different error rate than Bonferroni or Holm: it controls the expected proportion of false rejections among discoveries (the FDR), not the chance of any false rejection (FWER). The step-up scaling p * m / rank is less aggressive than Bonferroni's p * m, which is why BH typically rejects more hypotheses at the same threshold. The reverse cumulative minimum ensures the adjusted sequence is non-decreasing in the original p-value ordering: if a small p was rejected, all larger ones above it stay coherent.

Explanation: Only the trt1 vs trt2 contrast survives at FWER 0.05 (adjusted p = 0.009). The unadjusted p-value for that pair is roughly 0.003, and Bonferroni multiplies by 3 (the number of pairwise comparisons), giving 0.009. The marginal ctrl vs trt2 contrast moves from an unadjusted 0.029 up to an adjusted 0.087 and now fails the cutoff. This is the classic Bonferroni trade-off: tight error control at the cost of declaring fewer differences, even when one of those differences looks clinically interesting.

Explanation: All three pairs come up significant at alpha 0.05 after Holm correction. The setosa contrasts are wildly far from zero; the interesting case is versicolor vs virginica at 0.0018. Under Bonferroni that same contrast would be roughly 0.0018, since with m=3 the largest sorted p is multiplied by 1 under Holm but by 3 under Bonferroni, then the cummax kicks in only for the smaller ones. For this dataset both methods give nearly identical decisions; the gap widens as m grows.

Explanation: Both methods reject the same single hypothesis here because the gap between the smallest p (0.001) and the next (0.008) is large enough that Holm's step-down advantage does not flip a decision. Look at row 2: Bonferroni multiplies by 8 giving 0.064, while Holm multiplies by 7 giving 0.056. Neither clears 0.05. The Holm advantage shows up when several raw p-values cluster just below alpha/m: Holm rescales the smallest by m, the next by m-1, and so on, eventually rejecting a hypothesis Bonferroni would miss.

Explanation: Almost all 50 planted signals are recovered even at the strict FDR 0.01 threshold because they were drawn from Uniform(0, 0.01). Relaxing the threshold to 0.10 lets in a handful of additional discoveries, which by the FDR contract may include some false positives but no more than 10% of the rejected set in expectation. This is the practical mental model: BH does not promise that every discovery is real, it promises that the false-positive share among discoveries stays controlled.

Explanation: Benjamini-Yekutieli scales BH by a factor of sum(1/k) for k = 1..m, which for m = 500 is roughly 6.79. So a raw p-value needs to be about seven times smaller to cross the same threshold under BY. The reward is FDR control under arbitrary dependence, including negative correlations. Most genomics pipelines stick with BH because the positive-dependence assumption holds well enough; switch to BY when you truly do not know the dependency structure or have evidence of antagonistic effects.

Explanation: Read across each row to see the conservativeness ranking: Bonferroni and BY are the strictest, BH is the loosest, and Holm/Hochberg/Hommel sit between. Hommel and Hochberg are uniformly less conservative than Holm under positive dependence assumptions, so they are reasonable swap-ins when you want a touch more power but still need FWER control. Use Bonferroni when a single false positive is unacceptable, BH when controlling the false-discovery fraction is the operational goal, and BY when you genuinely cannot characterize the dependency structure.

Explanation: Diet 3 differs significantly from Diet 1 (adjusted p = 0.014), while every other pair fails to clear 0.05. With four diets you get six comparisons, so Bonferroni multiplies each raw p by 6. The diet-4-vs-diet-1 contrast (raw p around 0.0095) gets pushed up to roughly 0.057 and just misses the cutoff. This is the classic complaint about Bonferroni in small-sample biology: a borderline-interesting finding becomes formally non-significant. Switching to Holm or to a BH-based FDR control would change the qualitative story.

Explanation: All three pairs are crushingly significant because petal length separates the three iris species almost perfectly. The point of the exercise is the engineering: pairwise.wilcox.test() takes the same correction-method dispatch as pairwise.t.test(), so swapping a parametric test for a rank-based one does not change how you reason about multiplicity. Set exact = FALSE to get the asymptotic normal approximation when groups have ties, which suppresses the noisy warnings without changing the substantive conclusion at these sample sizes.

Explanation: TukeyHSD uses the studentized range distribution and is the proper FWER-controlling tool for ANOVA all-pairs contrasts, while Bonferroni is generic and ignorant of the joint distribution. Notice they disagree on the trt2-ctrl pair: Tukey gives 0.198 but Bonferroni gives 0.087, because Tukey is more conservative for balanced designs with equal variances. Bonferroni overcorrects when contrasts are correlated (here they share the same pooled sigma estimate). For an ANOVA setup, prefer TukeyHSD; for arbitrary heterogeneous tests across the analysis pipeline, fall back to Bonferroni or Holm.

Explanation: Under the null, p-values are uniformly distributed on [0, 1], so the unadjusted count near 50 (about 5% of 1000) is exactly what theory predicts. Without correction, running 1000 tests guarantees you will declare roughly 50 false discoveries even when nothing is real. Bonferroni multiplies every p-value by 1000, so a raw p of 0.0001 is needed to clear the threshold. The simulation gives you a vivid mechanical proof that family-wise error control collapses without adjustment as m grows.

Explanation: The realized false-discovery proportion (fdp) is 0.031, well below the nominal 0.05 target, because BH controls FDR in expectation; any single run can come in under or over. Discoveries (129) are well below the 200 true signals because effect size 0.8 with n = 30 yields modest power per test. Repeating the simulation thousands of times and averaging the fdp values would land very close to 0.05; that ensemble-average behavior is exactly what FDR control means. Single-run variability is normal.

Explanation: This is the canonical volcano-plot filter: significance from the adjusted p-value and biological relevance from the fold change. Either filter alone is a bad call: a gene with a tiny padj but |log2fc| < 0.1 is statistically real but biologically trivial, while a gene with a huge fold change but padj > 0.05 is likely chance. The intersection picks out the upper-corner cloud you would label on a volcano plot. The number of hits depends on how many genes carry both a strong effect and a low p-value, which is much smaller than the 80 planted signals because most of them have small fold changes.

Explanation: Bonferroni's empirical FDP is essentially zero, far below its FWER target of 0.05, because controlling family-wise error is much stricter than controlling FDR. BH's empirical FDP averages right near the nominal 0.05 target, exactly as the theory predicts under independence. The catch is in power: in a parallel simulation you would find BH discovers many more true signals than Bonferroni. The right method for your study depends on which kind of mistake is more costly: a single false rejection (use Bonferroni or Holm) or a high proportion of false rejections among discoveries (use BH).

Explanation: BH consistently finds more true signals than Bonferroni at every non-trivial effect size, which is the reason it dominates genomics and high-throughput screening pipelines. At effect = 0.6, BH finds 15 signals versus Bonferroni's 1; at effect = 1.0, the gap is 42 vs 18. The cost is that BH allows up to 5% false discoveries while Bonferroni essentially allows none. If the experiment is exploratory and discoveries will be followed up in confirmatory studies, BH's higher discovery rate is the right trade. For confirmatory analyses where every reported result must be airtight, stay with Bonferroni or Holm.

Explanation: After adjusting for the seven simultaneous coefficient tests, no term survives at alpha 0.05 under any method, even though wt has a raw p of 0.018 that looks compelling in isolation. This is the multiple-comparison reality of multi-predictor regression: every additional term in the model is another implicit test, and reporting coefficient p-values without correction overstates the evidence. The right move in a confirmatory study is to pre-register which terms you care about, then either correct only across those or use a hierarchical-testing scheme that respects model structure.