Navigate
Roadmap Tutorials Exercises Tools Get certified
Lesson 7 of 7

Compare many models with workflowsets

In Lesson 6 you tuned a single decision tree on the lender's loan book and confirmed it once on a sealed test set. But a tuned tree is only as good as the fact that it was a tree. In a real project you rarely commit to one model up front. You line up a few honest contenders, a plain logistic regression, that decision tree, a random forest, and you let them race.

The trick is making the race fair: every contender judged on the exact same folds, by the exact same metric, with no contender peeking at data the others did not. The chart below is the finish line, three models, one ROC AUC bar each, the tallest one wins. This lesson is how you produce that chart with one tool instead of three copy-pasted scripts.

By the end of this lesson you will be able to:

  • Explain why a fair model comparison needs the same resamples and the same metric for every contender
  • Build a workflow set that crosses your preprocessors with your model specs into one object
  • Run the whole bake-off across shared folds in one call, rank the leaderboard, and finalize the winner

Prerequisites: you can run R and use the |> pipe, and from earlier lessons you can bundle a recipe and model into a workflow (Lesson 3), score across cross-validation folds (Lesson 4), [measure with roc_auc](Measure-with-yardstick.html) (Lesson 5), and tune a model honestly (Lesson 6). We reuse that exact loan book here.

Why you cannot just eyeball it

One race, same rules for everyone

Here is the tempting, wrong way to compare three models: fit each one however is convenient, glance at a number for each, keep the biggest. It breaks in two quiet ways. First, if each model is scored on a different random split of the data, you are comparing how lucky each split was, not how good each model is. Second, if you score a model on the rows it trained on, the most flexible model always looks best because it can memorize, which is the overfitting trap you have met all course.

The fix is the same discipline you have built up lesson by lesson: judge every contender on one shared set of cross-validation folds with one shared metric. Same questions, same judges, same answer key. That is the entire idea behind a workflow set.

So we start exactly where Lesson 6 ended, the same loan book, the same split, the same five folds. Run this once to rebuild it in this session:

RInteractive R
library(rsample) set.seed(7) n <- 240 loans <- data.frame( income = round(runif(n, 22000, 98000)), # annual income, dollars age = round(runif(n, 21, 60)), employed = round(runif(n, 2, 160)), # months at current job home = factor(sample(c("own", "rent", "mortgage"), n, TRUE)) ) risk <- with(loans, plogis(-1.1 + 1.4 * (income < 45000) + 1.0 * (employed < 20) + 0.6 * (home == "rent"))) loans$defaulted <- factor(ifelse(runif(n) < risk, "yes", "no")) loans$defaulted <- relevel(loans$defaulted, ref = "yes") # "yes" is the event we predict set.seed(1) split <- initial_split(loans, prop = 0.75, strata = defaulted) train <- training(split) # 179 rows to compare on test <- testing(split) # 61 rows locked away for the final check folds <- vfold_cv(train, v = 5, strata = defaulted)

  

  





Warning
The test set stays sealed for the whole bake-off. Every model is compared using cross-validation on the training folds only. The locked test set is touched exactly once, at the very end, to confirm the single model you crowned. Use it to choose, and it stops being an honest estimate of new-data performance.