Ridge Regression and Shrinkage
Lesson 3 fit lines to the median and the tails. But notice what every method so far, ordinary and robust and quantile alike, has quietly assumed: that the data can pin down each coefficient in the first place. When your predictors are strongly correlated, it cannot, and least squares starts reporting nonsense with total confidence.
Meet Maya, a marketing analyst at a coffee-subscription company. Every week she predicts new signups from what she spent on six ad channels: search, social, display, video, email and affiliate. The catch is that she scales all six from one weekly budget, so in a busy week every channel is up and in a quiet week every channel is down. The six spend columns are almost the same column. When Maya fits ordinary least squares, it hands her a model claiming that display and email ads REDUCE signups, and the numbers lurch every time she refits on a slightly different stretch of weeks.
Ridge regression is the fix. By gently shrinking the coefficients toward zero, it trades a tiny, deliberate bias for a large drop in that wobble, and gives Maya a model she can trust. Drag the penalty slider below to watch coefficients shrink (switch to the Ridge (L2) toggle, the subject of this lesson).
By the end of this lesson you will be able to:
- Explain why strongly correlated predictors make ordinary least squares coefficients unstable, and spot the symptom
- Define ridge regression as penalized least squares, and say exactly what the penalty controls
- See how shrinkage trades a little bias for a large drop in variance, and why that can predict better
- Fit ridge in R, choose the penalty by cross-validation, and know when to reach for it
Prerequisites: you can fit and read a linear regression with lm() (a coefficient is a predictor's effect; a residual is actual minus predicted), and you have met the bias-variance tradeoff. If lm() is rusty, see linear regression. Lessons 1 to 3 of this course are useful context but not required.
The predictors move together
Let us build Maya's data so the problem is concrete. Each of 60 weeks has a spend figure for the six channels; because they all scale from one budget, they rise and fall together. Signups truly depend on just two of them, search and social, plus a little noise. (We build the data right here, since each lesson runs in a fresh R session.)
Read that correlation matrix: every pair of channels sits around 0.97, almost perfectly in step. Statisticians call predictors this tangled multicollinear, and the heatmap below makes it plain, a near-solid block of dark green. To ordinary least squares, these six columns carry nearly the same information six times over.