Navigate
Roadmap Tutorials Exercises Tools Get certified
Lesson 5 of 7

Measure with yardstick

In Lesson 4 you handed the risk committee a number: the loan-default model scores 0.648 accuracy, cross-validated, give or take a little. It felt solid. Then someone at the table tries something lazy: a "model" that simply stamps no default on every single applicant. On this loan book that scores about 63% too, higher than a fair few real models, and it catches exactly zero defaulters. If a do-nothing rule can match your model on accuracy, then accuracy was never the yardstick that mattered.

This lesson is about choosing the yardstick on purpose. The yardstick package gives you a whole toolbox of metrics; the skill is knowing which one answers your question.

By the end you will be able to:

  • Read a confusion matrix and say why accuracy alone can flatter a useless model
  • Compute precision, recall and specificity, and pick the one that matches the business cost
  • Tell a hard-class metric from a probability metric, and read an ROC curve and its AUC
  • Bundle your chosen metrics into a metric set and read them across every resample

Prerequisites: you can run R and use the |> pipe, and you can bundle a recipe and a model into a workflow and resample it (Lesson 4).

The trap

When accuracy flatters a useless model

Here is the lender's loan book again, rebuilt right here so this page runs on its own. Each row is one applicant; defaulted is what we are trying to predict. We set the factor levels so that yes (a default) comes first, because that is the outcome the bank cares about catching.

RInteractive R
library(dplyr) set.seed(7) n <- 240 loans <- data.frame( income = round(runif(n, 22000, 98000)), # annual income, dollars age = round(runif(n, 21, 60)), employed = round(runif(n, 2, 160)), # months at current job home = factor(sample(c("own", "rent", "mortgage"), n, TRUE)) ) risk <- with(loans, plogis(-1.1 + 1.4 * (income < 45000) + 1.0 * (employed < 20) + 0.6 * (home == "rent"))) loans$defaulted <- factor(ifelse(runif(n) < risk, "yes", "no"), levels = c("yes", "no")) table(loans$defaulted) #> #> yes no #> 89 151

  

  





Of 240 applicants, 89 defaulted and 151 did not. The classes are lopsided: only about 37% of applicants are defaulters. That imbalance is exactly what lets accuracy lie. Watch what the do-nothing rule scores:




  

    
RInteractive R

    
mean(loans$defaulted == "no")   # accuracy if we predict "no default" for everyone
#> [1] 0.629

    

      
      
    

    

  

  





Predicting "no default" for everybody is right 62.9% of the time, purely because most applicants really do repay. It is a model with zero skill and a respectable-looking score. Now fit a genuine model, a logistic regression, on a training split and measure its accuracy on held-out applicants:




  

    
RInteractive R

    
library(rsample)
library(parsnip)
library(yardstick)

set.seed(1)
split <- initial_split(loans, prop = 0.75, strata = defaulted)
train <- training(split)
test  <- testing(split)

logit_spec <- logistic_reg() |> set_engine("glm") |> set_mode("classification")
fit        <- logit_spec |> fit(defaulted ~ ., data = train)

preds <- augment(fit, new_data = test)   # adds .pred_class, .pred_yes, .pred_no
accuracy(preds, truth = defaulted, estimate = .pred_class)
#> # A tibble: 1 x 3
#>   .metric  .estimator .estimate
#>   <chr>    <chr>          <dbl>
#> 1 accuracy binary         0.656

    

      
      
    

    

  

  





The real model scores 0.656. It beat the do-nothing rule (0.629) by barely two points. If you reported only that number, the committee would think the model is roughly as good as guessing "no" every time. Accuracy is not wrong here, it is just answering the wrong question. To see what the model is really doing, we have to look at which applicants it gets right and wrong.



Note
augment() takes a fitted model and a data frame and adds the model's predictions as new columns: .pred_class (the predicted label at the usual 0.5 cutoff) and .pred_yes / .pred_no (the predicted probability of each class). Every yardstick metric reads one or more of those columns.