Cross Validation Exercises in R: 20 Real-World Practice Problems

Explanation: rep(1:5, length.out = n) builds a balanced fold vector that recycles the IDs in order; sample() then permutes it so the folds are random but still balanced. Using sample(1:5, 32, replace = TRUE) would be wrong: it draws independently, so fold sizes drift away from n/k and one fold can easily be twice the size of another. Always pair fold construction with set.seed() for reproducibility.

Explanation: A single holdout split gives one estimate of generalization error and is fast, but the variance is huge on a 32-row dataset. The exact RMSE you observe depends entirely on which 23 rows ended up in train. That is why k-fold CV (next section) averages over multiple splits: the mean over folds is what you trust, not any single split. createDataPartition stratifies on the outcome quantiles, which keeps the train/test mpg distributions similar.

Explanation: LOOCV uses every row as a test point exactly once, so there is no randomness and no seed is needed: it always returns the same number. The cost is n model fits. For ordinary least squares there is a closed-form shortcut using the hat matrix that avoids the loop entirely; running through for here is a warm-up so the next exercise (k-fold) is just a generalization of the same skeleton.

Explanation: LOOCV uses 31 rows for every fit instead of 22, so its training set is closer to the full dataset and its bias is smaller. On a 32-row dataset that bias dominates: the holdout estimate 3.01 is upward biased because the fit only saw 22 rows. The trade-off LOOCV pays is higher variance: each of the 32 fits sees almost the same data, so the 32 errors are correlated. K-fold with k=5 or k=10 is the practical compromise.

Explanation: Pooling all 32 out-of-fold predictions into a single RMSE is the standard "stacked" version of k-fold and is what most papers report. The alternative is computing RMSE inside each fold and averaging the 5 values: that gives a slightly different number because the folds have unequal sizes, and the unweighted mean would over-weight small folds. The stacked version is mathematically a weighted mean by fold size, so it is the safer default.

Explanation: Repeated k-fold reduces the variance of the CV estimate by averaging over the random fold assignment. Look at the spread of the 10 values: roughly 2.68 to 2.91. Any one fold split could mislead you by 0.1 RMSE, which is more than the gap between many candidate models. The conventional recipe is 5-fold or 10-fold, repeated 5 to 10 times. For small datasets like mtcars, repeats matter most; for large datasets a single 10-fold is usually enough.

Explanation: Stratification matters when class proportions are imbalanced or when the outcome is rare: a non-stratified split could leave an entire fold without a positive class, breaking AUC computation. createFolds stratifies on the supplied vector; for regression it bins the numeric outcome into quartiles and stratifies on the bins. Tidymodels' rsample::vfold_cv(strata = Species) does the same thing and is the preferred API for new code, but createFolds is still the simplest one-liner when you just need fold IDs.

Explanation: train() is caret's single dispatch for fitting any of 200+ model engines under a unified resampling protocol. The method = "lm" call has no tuning parameters, so the 10-fold result is just a vanilla 10-fold CV summary. The same call with method = "rf" would tune mtry over a default grid using the same 10 folds: that is the design that makes caret useful even when you outgrow basic lm.

Explanation: repeatedcv runs the entire 10-fold protocol repeats times with a fresh fold randomization each time, giving you a more stable estimate of the RMSE for each candidate k. The RMSESD column reports the standard deviation across the 50 fold-level RMSE values, which feeds the one-standard-error rule in later exercises. Always pair kNN with preProcess = c("center", "scale") because distance metrics are scale-dependent: a 100x range column will dominate the neighbor calculation.

Explanation: Leave-group-out (also called Monte Carlo CV or repeated random subsampling) is k-fold's looser cousin: every resample is an independent draw, so rows can appear in many test sets and others in none. It is useful when you want a tunable train fraction (the p argument) and a large number of resamples for variance estimation. Its weakness is that test sets overlap, so the resample errors are positively correlated and confidence intervals are narrower than they should be.

Explanation: Classification CV is usually wrong on the first try because the default summaryFunction returns Accuracy and Kappa, and the default metric = "Accuracy" is misleading on imbalanced data. The three steps to get correct ROC-AUC out of caret are: set classProbs = TRUE, set summaryFunction = twoClassSummary, and set metric = "ROC" on the train() call. Skip any of the three and you silently get an Accuracy-driven model instead of a ranking-driven one.

Explanation: rsample represents each fold as an rsplit object that holds row indices, not data: actual subsetting happens lazily through analysis() (the training partition) and assessment() (the held-out partition). This keeps a 10-fold object lightweight even on large data, and it pairs cleanly with purrr::map() to fit one model per split without duplicating the dataset 10 times. Tidymodels' workflows build on this abstraction.

Explanation: The map-over-splits pattern is the tidy alternative to caret::train: explicit, composable, and works with any modelling function whose interface is fit + predict. Note the mean of per-fold RMSEs (2.83) is not identical to the pooled RMSE from Exercise 2.1 (2.75) because fold sizes are unequal: the unweighted mean over folds overweights the small folds. Both are valid CV estimates; just be consistent within a project.

Explanation: Repeated-measures data leaks when you split rows naively: visit 3 of patient P1 in train and visit 4 of P1 in test means the model effectively memorizes the patient. group_vfold_cv() enforces that all rows from the same group land in the same partition. The same principle applies to time-grouped data (split by week, not by row), user-grouped data (split by user, not by event), and any clustered design. Skipping this step is the most common cause of "great CV, terrible production" outcomes.

Explanation: Standard k-fold is wrong for time-series because it lets future data train a model that predicts the past. rolling_origin() slides a fixed-width training window forward and assesses on the immediately following block. The skip argument controls the stride between successive splits; skip = 5 means each new origin advances 6 months (skip + 1) so the assessment windows do not overlap. For sales, energy, or sensor data this is the only honest CV.

Explanation: The fixed-window version forgets old data; the expanding-window version remembers everything. Pick fixed when the underlying process drifts (consumer behavior, financial regimes) so old data hurts. Pick expanding when the process is stationary and more data always helps. A useful diagnostic is to fit both and compare CV error: if expanding is much better, the series is stationary; if fixed is much better, recent data is more representative.

Explanation: The stacked error tibble is the single most useful artifact from time-series CV: from it you can derive RMSE by split, RMSE by horizon (1-step, 2-step, etc.), residual autocorrelation, and a forecast plot. Once you have this tibble, swap lm() for forecast::auto.arima() or prophet::prophet() without changing the surrounding map skeleton. The forecast pipelines all share this shape because the rolling-origin contract is model-agnostic.

Explanation: Identical seeds give identical fold assignments across engines, so the RMSEs compare apples to apples; without seed control, the kNN tune might land on a friendlier fold split than the lm fit. On a 32-row dataset like mtcars, simple linear regression usually wins because tree and kNN methods are starving for data. The lesson generalizes: try the simplest model first and only justify complexity when CV says it pays.

Explanation: The one-SE rule says "pick the simplest model whose CV error is within one standard error of the best": you trade a touch of accuracy for a meaningfully simpler model, often avoiding overfitting on the validation set itself. The standard error of the mean over 50 fold-level RMSEs is RMSESD / sqrt(50), not RMSESD. For kNN "simpler" means larger k; for trees it means deeper pruning (smaller tree); for lasso it means more shrinkage.

Explanation: Plain CV with hyperparameter tuning gives an optimistically biased error estimate: the same folds that picked the winning k also report the error, so the winner benefits from a lucky fit. Nested CV separates the two concerns: an inner CV picks k from the training partition only, and the outer fold reports the test error on data the tuner never saw. The cost is outer_folds * inner_folds * inner_repeats * grid_size fits (here: 5 10 3 * 10 = 1500), which is why it is reserved for final benchmarking rather than rapid iteration.