Navigate
Courses Tutorials Practice Pricing Tools Upgrade
Lesson 1 of 7

Cross-Validation Strategies

You already know the golden rule of honest evaluation: never judge a model on the data it trained on. Hold out a test set, and its score stands in for how the model will do on the future. But there is a crack in that plan.

Sam, an analyst at a used-car marketplace, is checking a model that predicts a car's fuel economy from its weight and horsepower. He holds out a quarter of his cars, scores the model, and gets a typical error of 3.3 mpg. He reshuffles, holds out a different quarter, and gets 1.6 mpg. Same cars, same model, and the verdict just doubled. Which number does he report? Cross-validation is the fix, and this lesson builds it straight out of that problem.

By the end you will be able to:

  • Explain why a single train/test split gives a luck-of-the-draw score
  • Run k-fold cross-validation and read its result
  • Weigh LOOCV and repeated k-fold, and choose a sensible number of folds

Prerequisites: you can fit a model with lm() and call predict(), and you know what a held-out test set is (Lesson: Train, Validation, Test, and Data Leakage).

The problem

One split is a coin flip

Let us watch Sam's problem happen for real. The data is mtcars: 32 cars that Motor Trend road-tested in 1974, each with its fuel economy (mpg), weight (wt, in thousands of pounds) and horsepower (hp).

RInteractive R
head(mtcars[, c("mpg", "wt", "hp")]) #> mpg wt hp #> Mazda RX4 21.0 2.620 110 #> Mazda RX4 Wag 21.0 2.875 110 #> Datsun 710 22.8 2.320 93 #> Hornet 4 Drive 21.4 3.215 110 #> Hornet Sportabout 18.7 3.440 175 #> Valiant 18.1 3.460 105

  

  





Sam holds out 8 of the 32 cars (a quarter), fits an ordinary regression on the other 24, and measures the typical prediction error on the 8 he held out. That error is the RMSE (root-mean-square error), reported in mpg: an RMSE of 2.5 means the model is off by about 2.5 mpg on a typical car. Then he does it again with a different random 8, and again, six times over:




  

    
RInteractive R

    
rmse_of <- function(actual, predicted) sqrt(mean((actual - predicted)^2))

split_rmse <- function(seed) {
  set.seed(seed)
  holdout <- sample(nrow(mtcars), 8)                 # 8 cars set aside to test on
  fit     <- lm(mpg ~ wt + hp, data = mtcars[-holdout, ])
  rmse_of(mtcars$mpg[holdout], predict(fit, mtcars[holdout, ]))
}
round(sapply(1:6, split_rmse), 2)                    # six different random hold-out sets
#> [1] 2.52 3.28 1.60 2.23 2.36 1.99

    

      
      
    

    

  

  





Every number came from the same 32 cars and the same model. The only thing that changed was which 8 cars happened to land in the hold-out set, yet the estimated error swings from 1.60 to 3.28 mpg, more than double. With only 8 cars deciding the score, one unlucky pair of gas-guzzlers can wreck it. A single split does not really measure the model; it measures the model plus the luck of the draw.