Reproducible pipelines with targets

You can build a model. Now you have to ship it, and keep it trustworthy. Meet Dev, a data scientist at a meal-kit company. He has one script, analysis.R, that predicts which customers will cancel: it reads the raw orders, aggregates them into per-customer numbers, fits a model, and writes a summary table. Start to finish it takes about 40 minutes, because fitting the model is slow.

This afternoon Dev changed the summary table six times, each time to round a number differently. Six times he waited 40 minutes. This lesson is about the tool that ends that: targets, which reruns only the steps that actually changed.

By the end of this lesson you will be able to:

• Explain why one long analysis script is slow to iterate on and lets your results quietly drift out of sync with your code
• Describe a targets pipeline as steps wired into a dependency graph, where each step is the output of a function
• Predict exactly which steps rerun after a change, so a one-character edit costs seconds instead of 40 minutes

Prerequisites: you can fit a model end to end, you can write a function, and it helps to have met reproducibility with renv and git. The four boxes below are Dev's whole analysis.

One script reruns everything

Dev's analysis lives in a single file. To run it, he calls source("analysis.R"), and it executes top to bottom:

# analysis.R - the whole analysis in one file
library(readr); library(dplyr)

orders   <- read_csv("orders.csv")   # 12,000 raw order rows
features <- get_features(orders)     # aggregate to 800 customers
model    <- fit_model(features)      # trains the model ... ~38 minutes
summary  <- summarize_model(model)   # the little table at the end
write_csv(summary, "summary.csv")

The trouble is that source() is all-or-nothing. Change the last line, the one that rounds a number in the summary, and R still reruns the 38-minute model fit above it, because it has no idea the model did not change. Every tiny edit costs the full 40 minutes.

So Dev does what everyone does under time pressure: he comments out the slow line and reuses the model object still sitting in his workspace from the last run. It is faster, but now something dangerous has happened. The summary.csv he ships was built from an old model, while the code on disk describes a new one. The results and the code have silently drifted apart, and nobody can tell.