The Resampling 15: Bootstrap and Permutation Problems in R

Explanation: Each sample(x, replace = TRUE) draws 32 values from the same 32, so some appear twice and others not at all, and that is exactly the variation the bootstrap uses to mimic drawing fresh samples from the population. The standard deviation of the 2000 resample means is the bootstrap standard error. Because it assumes no particular distribution, the same recipe works for statistics that have no tidy closed-form standard error. Setting the seed first is what makes your number reproducible.

Explanation: The median is a rank-based statistic, so classical standard-error formulas either do not exist or lean on shaky assumptions. The bootstrap sidesteps all of that by recomputing the median on thousands of resampled datasets and measuring how much it moves. Notice the bootstrap standard error here is larger than the mean's, which reflects the median hopping between the two central mpg values as the resample composition changes. The principle is general: any statistic you can compute, you can bootstrap.

Explanation: The key move is resampling rows, not columns. If you sampled mpg and wt independently you would shatter their pairing and bootstrap a correlation near zero. Drawing one index vector and applying it to both columns keeps each car's (mpg, wt) pair together, so the resampled correlation reflects the real relationship. The small bootstrap standard error, about 0.03 against an estimate near -0.87, tells you the strong negative correlation is stable rather than an artifact of a few cars.

Explanation: The percentile interval is the simplest bootstrap confidence interval: sort the resampled statistics and cut off the bottom and top 2.5 percent. It needs no assumption that the sampling distribution is symmetric or normal, which is its main advantage over the textbook mean plus or minus 1.96 * se interval. The trade-off is that it can be biased for small samples or skewed statistics, where the bias-corrected and accelerated (BCa) interval does better. For a roughly symmetric statistic like this mean, the percentile interval lands close to the classical result.

Explanation: Moving from 95 to 90 percent confidence simply narrows the interval, because you are accepting a higher chance of missing the true value. The bounds here land on round numbers because the bootstrap median can only take values that actually appear in mpg, so its distribution is discrete and lumpy. That lumpiness is a genuine feature of bootstrapping an order statistic from a small sample, and it is why a percentile interval for a median looks chunkier than one for a mean.

Explanation: The bootstrap estimates bias by asking how far the average resampled statistic drifts from the original estimate. Here the bias is tiny, about -0.002, so the sample correlation is very nearly unbiased and no correction is warranted. When the bias is large relative to the standard error you can subtract it to form a bias-corrected estimate, though doing so inflates variance and is often not worth it. The correlation coefficient is mildly biased in general, and this drill shows the effect is negligible for this dataset.

Explanation: This is case resampling, or row resampling: each bootstrap dataset is a fresh sample of whole cars, so the refit slope varies because the mix of cars changes. The resulting standard error makes no assumption about the error distribution, which is why it is a useful cross-check on the summary(lm()) standard error. When the two disagree sharply, it usually signals heteroskedasticity or an influential point. An alternative is residual resampling, which holds the predictors fixed and resamples residuals, but case resampling is more robust when the mean model itself may be misspecified.

Explanation: The two intervals share an upper bound near -4.2 but the bootstrap reaches lower at the bottom, a sign the slope's sampling distribution is slightly left-skewed once you drop the normality assumption. confint() inverts a symmetric t interval, so by construction it cannot show that asymmetry. When bootstrap and normal-theory intervals line up you gain confidence the model assumptions are reasonable; when they diverge, trust the bootstrap, because it reads the actual resampling behavior instead of assuming a shape. The column labels come from quantile(), since rbind() keeps the first argument's names.

Explanation: Permutation differs from the bootstrap in one crucial way: it samples the labels without replacement, which is what enforces the null hypothesis of no group effect. The permutation mean sits essentially at zero, as it must when the labels are meaningless, and its standard deviation is the spread you would see by chance alone. The observed 7.2 mpg gap is more than three of those chance standard deviations away from zero, which already hints the difference is real. The next exercise turns that hint into a p-value.

Explanation: The add-one correction, $p = (1 + \#\{|d^*| \ge |d_\text{obs}|\}) / (B + 1)$, treats the observed data as one more equally likely arrangement, which keeps the p-value strictly positive and very slightly conservative. Here almost no shuffle beats the real 7.2 mpg gap, so the p-value is about 0.0005, strong evidence that transmission type and fuel economy are linked. Taking absolute values is what makes the test two-sided; drop it and you would be testing a one-sided alternative instead.

Explanation: This is a borderline result: at about 0.055 the p-value sits just above the conventional 0.05 line, so supplement type alone is not quite significant once you ignore dose. That is a useful lesson in itself, since real data rarely hands you a clean verdict, and a permutation test is honest about that ambiguity rather than hiding it behind a formula. Pooling across the two dose levels also dilutes the effect, because dose is the stronger driver of growth in this experiment. A permutation done separately within each dose would be the natural next step.

Explanation: Both tests agree on the substance: the transmission effect is highly significant either way. They differ in the small print because the t-test relies on a theoretical t distribution while the permutation test builds its null distribution directly from the data, so their p-values need not match digit for digit. With 32 observations and a large effect, the two land close together. The permutation test's real advantage appears when the t-test's normality or equal-variance assumptions fail, which is exactly what the final comparison explores.

Explanation: Neither test flags a significant difference, but they disagree sharply on how unlikely the data are. The outlier of 40 dominates both the mean and the variance of group_a, so the observed mean gap is small relative to the enormous spread. The permutation test sees that almost every reshuffle produces an equally extreme or more extreme gap once the 40 lands in either group, pushing its p-value near 1. The t-test's p-value near 0.53 is not trustworthy here, because the outlier wrecks the normality its formula assumes. The lesson: with heavy tails or outliers, prefer a resampling or rank-based test over the t-test.

Explanation: The jackknife standard error is $\sqrt{\frac{n-1}{n}\sum_{i}(\bar{x}_{(i)} - \bar{x}_{(\cdot)})^2}$, where $\bar{x}_{(i)}$ is the mean with observation $i$ removed and $\bar{x}_{(\cdot)}$ is the average of those leave-one-out means. That $(n-1)/n$ inflation factor is what turns the tiny leave-one-out variation into a proper standard-error estimate. For a smooth statistic like the mean it matches both the classical formula and the bootstrap almost exactly, which is a reassuring sanity check. The jackknife is fully deterministic, so the seed here changes nothing; it is set only to keep every drill in this collection uniform.

Explanation: Each leave-one-out correlation asks how much the estimate depends on a single car, and the one that moves it most, the Chrysler Imperial, is the most influential point. The jackknife bias, $(n-1)(\bar{\theta}_{(\cdot)} - \hat{\theta})$, is again tiny here, confirming the correlation is stable. Influence diagnostics like this are the jackknife's real strength: leaving each point out one at a time exposes fragile estimates that a single number could hide. When one observation dominates, that is a cue to inspect it before trusting the fit.