Navigate
Roadmap Tutorials Exercises Tools Get certified
Lesson 4 of 7

Resample with rsample

The risk committee has one question about your loan-default model: how accurate is it, really? In Lesson 3 you bundled the recipe and the model into a workflow, split the 240 applications 75/25, scored the held-out quarter, and got 82% accuracy. You write that down. Then a colleague reshuffles the very same data, takes a fresh 75/25 split, and gets 76%. Same model, same applicants, same code. Which number do you put in the report?

The honest answer is neither, because a single split rests its whole verdict on one slice of luck. This lesson replaces that one coin-flip with resampling: score the model on many splits and read the average, with a measure of how much it wobbles. The rsample package builds those splits for you.

By the end of this lesson you will be able to:

  • Explain why one train/test split gives an unreliable estimate of how good a model is
  • Build cross-validation folds and bootstrap resamples with rsample
  • Run your whole workflow across every resample and read the average score and its spread

Prerequisites: you can run R and use the |> pipe, and you can bundle a recipe and a model into a workflow and fit it (Lesson 3).

The problem

One split is a roll of the dice

Let us prove the wobble rather than assert it. Here is the lender's loan book again, rebuilt right here so this page runs on its own. Each row is one applicant; defaulted is what we are trying to predict.

RInteractive R
set.seed(7) n <- 240 loans <- data.frame( income = round(runif(n, 22000, 98000)), # annual income, dollars age = round(runif(n, 21, 60)), employed = round(runif(n, 2, 160)), # months at current job home = factor(sample(c("own", "rent", "mortgage"), n, TRUE)) ) risk <- with(loans, plogis(-1.1 + 1.4 * (income < 45000) + 1.0 * (employed < 20) + 0.6 * (home == "rent"))) loans$defaulted <- factor(ifelse(runif(n) < risk, "yes", "no")) table(loans$defaulted) #> #> no yes #> 151 89

  

  





Now fit the same logistic model on twenty different random 75/25 splits and record the accuracy each time. Nothing about the model changes between runs. Only the luck of which rows land in the test set changes.




  

    
RInteractive R

    
acc_of_one_split <- function(seed) {
  set.seed(seed)
  i  <- sample(nrow(loans), 0.75 * nrow(loans))   # pick 180 rows to train on
  tr <- loans[i, ]; te <- loans[-i, ]             # the other 60 are the test set
  m  <- glm(defaulted ~ ., data = tr, family = binomial)
  p  <- ifelse(predict(m, te, type = "response") > 0.5, "yes", "no")
  mean(p == te$defaulted)                         # accuracy on the held-out rows
}
accs <- sapply(1:20, acc_of_one_split)
round(range(accs), 3)   # the lowest and highest accuracy we saw
#> [1] 0.567 0.750
round(sd(accs), 3)      # how much it swings from split to split
#> [1] 0.053

    

      
      
    

    

  

  





Read that again. The identical model scored as low as 57% and as high as 75%, purely because of which applicants happened to sit in the test set. If you had run it once and stopped, you would have walked into that meeting with a number that was off by up to nine points in either direction, and no way to know it.