Your First End-to-End Model in R

Three lessons of groundwork come together here. You learned to frame a business question as a prediction task (Lesson 1), why a more flexible model can still fail on new data (Lesson 2), and how to set aside honest data with no leaks (Lesson 3). Now you build the whole thing: one model, start to finish.

Remember Priya from Lesson 1? She runs retention at FreshBox, the meal-kit subscription, and wanted to catch customers who are about to cancel so a small win-back offer reaches only them. Back then we only framed her problem. Today we actually build her model: split, fit, predict, evaluate, and make every number reproducible.

By the end of this lesson you will be able to:

• Run a complete workflow on one dataset: split, fit, predict, evaluate
• Fit a classifier on the training rows and score customers it has never seen
• Judge the model with a confusion matrix and the metric that matches Priya's decision, not just accuracy
• Make the entire run reproducible with a fixed seed and one script

Prerequisites: you can run R and read its output, and you have the three earlier lessons: Framing a Problem as ML, The Bias-Variance Tradeoff, and Train, Validation, Test, and Data Leakage.

The five stages below are the whole journey. We walk through each one on Priya's real data.

One table, and a seed before anything else

Priya's raw data is one table: one row per FreshBox customer, holding the handful of facts she knows at decision time. Each lesson here runs in a fresh session, so we build that table inline. The very first line is set.seed, before a single random number is drawn, because reproducibility starts at the first random call, not at the end.

Nine hundred customers, of whom 257 churned. The four features are the classic customer-health signals: how recently someone ordered, how often, how much, and how long they have been a subscriber. churned (yes or no) is the target Priya wants to predict. That last column is decided by a rule we will never hand the model, exactly as real churn is decided by the world, not by us.

What set.seed actually does

A computer's "random" numbers are not truly random: they come from a formula with a starting point. set.seed fixes that starting point. Give it the same number and you get the same sequence of draws, every time, on any machine. Watch:

Same seed, same four numbers. Change the seed and the draw changes. This is why every reproducible analysis pins its seed: it turns "random" into "random but repeatable," so your result is something a colleague can recreate exactly rather than merely something close.

What makes a run reproducible?

Priya emails her script to a teammate. He runs it and gets slightly different coefficients and a different recall every single time he reruns it. What one change makes every run land on identical numbers?

Before you model, look

Good practice is to eyeball the data before fitting anything. This is only a look, not a decision: we are not choosing features or transforms from it, so nothing about the test customers leaks into the model. We just want to know whether there is any visible signal to learn.

Each dot below is a FreshBox customer. The horizontal axis is recency (days since their last order) and the vertical axis is frequency (orders in the last 90 days). The colour marks what actually happened: one colour for customers who churned, another for those who stayed. Press Run to draw the real chart.

The churned customers lean toward the lower right: more days since their last order, fewer recent orders. But the two groups overlap, plenty of stayers sit right among them. That overlap is the whole reason Priya needs a model: no single cutoff cleanly separates churn from stay, so we want something that weighs all four features together and outputs a graded probability instead of a hard rule.

Hold out an honest test set

Straight from Lesson 3: before we fit anything, we hide a slice of customers to judge the finished model on. The model trains as if those rows do not exist, and they become our stand-in for the future customers Priya will actually face. We hold out 30% for the test set and keep 70% to train on.

One quick sanity check: a good split keeps the same class balance in both halves, so the test set is a fair sample of the whole.

About 28% churn in each half. The split is honest and representative, and test will not be touched again until the very end.

A model that outputs a probability

Priya does not want a bare yes or no; she wants a churn probability for each customer, so she can rank who is most at risk. Logistic regression does exactly that, in two moves.

First it combines the four features into a single score, a weighted sum:

\[ z = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 + \beta_4 x_4 \]

where \(x_1,\dots,x_4\) are the four features (recency, frequency, basket, tenure), each \(\beta_j\) (beta) is a weight the model learns, and \(\beta_0\) is a baseline offset. A large positive \(z\) means high risk; a large negative \(z\) means low risk. But \(z\) can be any number, and a probability must sit between 0 and 1. So the second move squashes \(z\) through the logistic (sigmoid) function:

\[ P(\text{churn}) = \sigma(z) = \frac{1}{1 + e^{-z}} \]

where \(e \approx 2.718\) is Euler's number and \(\sigma\) (sigma) is the name of this S-shaped squashing function. When \(z = 0\), \(\sigma(0) = 0.5\); as \(z\) climbs, the probability rises toward 1, and as \(z\) falls, it drops toward 0. Drag the threshold on the curve below to see how a continuous probability becomes a yes or no call, and how the mistakes trade off as you move it.

Learn the weights on the training rows only

Fitting means letting the model choose the weights (\(\beta_0\) through \(\beta_4\)) that best match the training customers. In R, glm with family = binomial fits exactly the logistic model from the last step. We fit on train alone, never on test, so the test set stays the honest stranger from Lesson 3.

Read the signs, they tell Priya's story. days_since_last has a positive weight (+0.047): the longer since a customer's last order, the higher their churn risk. orders_90d (-0.197) and tenure_mo (-0.012) are negative: frequent, long-standing customers churn less. And avg_basket sits near zero (-0.004), so how much someone spends per order barely predicts whether they leave. The model discovered all of that from the training data alone.

Score the customers the model never saw

Now the honest test. We hand the fitted model the held-out customers and ask, for each one, its probability of churning. predict with type = "response" returns that probability; we then call anyone above a 0.5 cut a predicted churner.

The newdata = test argument is the whole game: we score the rows the model never trained on. Look at customer 2, the model gave them only a 9% churn chance, yet they actually churned. No model is perfect, and the next steps are about measuring exactly how often it is right, and in which direction it errs.

Predict one real customer

A single customer just tripped Priya's alarm: 45 days since their last order, only 2 orders in the last 90 days, a $60 average basket, and 10 months as a subscriber. Should the win-back coupon go out? Finish the line so predict scores this new customer with the fitted model. Fill in the blank.

new_cust <- data.frame(days_since_last = 45, orders_90d = 2, avg_basket = 60, tenure_mo = 10)
round(predict(fit, newdata = new_cust, type = "response"), 3)
#>     1 
#> 0.625

Score it: the confusion matrix

A single accuracy number hides the mistakes that matter. The confusion matrix shows all four outcomes at once: for the held-out customers, what the model said versus what actually happened.

Read it as four boxes. The model correctly cleared 181 stayers and correctly flagged 31 churners. It also made two kinds of error: 13 false alarms (customers it flagged who actually stayed) and, more painfully for Priya, 45 misses (churners it failed to flag). Three standard metrics summarise this, each answering a different question. Writing \(TP\) for the churners caught (31), \(FP\) for the false alarms (13), \(FN\) for the misses (45), and \(TN\) for the stayers correctly cleared (181):

\[ \text{accuracy} = \frac{TP + TN}{TP + TN + FP + FN}, \qquad \text{precision} = \frac{TP}{TP + FP}, \qquad \text{recall} = \frac{TP}{TP + FN} \]

Accuracy is the share of all calls that were right. Precision asks: of the customers we flagged, how many really churned? Recall asks: of the customers who really churned, how many did we catch? These are not the same question, and Priya cares far more about one of them.

Compute the recall yourself

Recall is the metric Priya lives by: of the customers who truly churned, what fraction did the model catch? From the confusion matrix cm, the churners caught are cm["yes", "yes"], and all real churners are the whole "yes" row, sum(cm["yes", ]). Fill in the denominator.

recall <- cm["yes", "yes"] / sum(cm["yes", ])
round(recall, 3)
#> [1] 0.408

Is 78.5% accuracy good?

That sounds fine, until you meet the laziest possible model: always predict "stays." Since most customers do stay, that alone scores surprisingly well.

The always-"stays" baseline is already 71.9% accurate, so our model's 78.5% is only a modest lift. Worse, its recall is 0.408: it catches fewer than half the customers who actually churn, and those are precisely the people Priya's coupon needs to reach. Accuracy flattered a model that is missing most of the churn.

The fix here is not a fancier model, it is a better threshold. We called anyone above 0.5 a churner. Lower that bar to 0.3, flagging every customer with at least a 30% chance of leaving, and recall climbs steeply.

Now the model catches 77.6% of churners, up from 41%. The price is precision, down to 0.484, so about half of Priya's coupons go to people who would have stayed anyway. For her that is a fine trade: a wasted coupon costs a couple of dollars, a lost customer costs far more. The threshold is a business dial, not a fixed law.

Which metric should Priya optimise?

A win-back coupon costs FreshBox about $3. A customer who churns unnoticed is worth hundreds in lost subscription revenue. Given those costs, which metric should Priya push hardest to improve?

Same seed, same answer

Every number in this lesson, the split, the coefficients, the recall, came from random draws. Without a fixed seed, rerunning the script would reshuffle the split and nudge every result. The seed makes the randomness repeatable, so you, Priya, and a colleague on another laptop all get identical numbers.

Two draws, same seed, provably identical. That is reproducibility in one line.

Priya's model, end to end, in one script

Here is everything you built, top to bottom, as the single reproducible script Priya would save. Read it as the five stages from the cover: seed, split, fit, predict, evaluate. Run it and you get her headline number, the recall at the threshold she chose.

One short script, and it never leaks, never fakes a number, and lands on the same 0.776 every single run. That is a complete, honest, reproducible model.

A first model, not the last word

Priya has a working, honest, reproducible model. It is a starting point, not the finish line, and a good data scientist says so out loud. Three limits are worth naming:

• One split gives a noisy score. Your test set was a single random 30% of customers. A different seed would hand you a slightly different accuracy and recall. The cure is cross-validation: rotate the held-out slice several times and average, for a steadier estimate. That is the heart of the next sections.
• Logistic regression draws a straight boundary (it is linear in the log-odds, the \(z\) from earlier). If churn depends on the features in a curvier way, a tree, a forest, or a boosted model may do better. The Classification and Boosting courses build those.
• One threshold is one policy. We picked 0.3 to favour recall. The right cut depends on the real cost of a missed churner versus a wasted coupon, a business decision to revisit as those costs change.

References

Four solid places to take this workflow further:

• An Introduction to Statistical Learning, ch. 2 and 4 (free PDF) - assessing model accuracy, and logistic regression as a classifier, at a gentle pace.
• The Elements of Statistical Learning, ch. 7 (free PDF) - model assessment and selection: the rigorous account of test error and why a held-out estimate matters.
• R for Data Science, 2nd edition (Wickham, Cetinkaya-Rundel, Grolemund) - the end-to-end data-analysis workflow in R, from raw data to a communicated result.
• tidymodels - the modern R framework that turns this hand-rolled pipeline into reusable, leak-safe steps (rsample splits, parsnip fits, yardstick metrics) you will use next.

Lesson 4 complete

You built a machine learning model end to end: you fixed a seed so the run is reproducible, split the data honestly, fit a classifier on the training rows, predicted on customers it had never seen, and judged it with the confusion matrix and the metric that fits Priya's decision. Then you packaged the whole thing into one script that anyone can rerun to the same numbers.

That is the entire Machine Learning Workflow course. You now have the skeleton every project shares. From here, the model-specific courses go deeper into the pieces you just used: Regression digs into fitting continuous outcomes, Classification unpacks logistic regression and its rivals, and both lean on cross-validation to turn today's single, noisy test score into a steadier estimate. Same five stages, sharper tools.