Navigate
Roadmap Tutorials Exercises Tools Get certified
Lesson 2 of 3

From one tree to a forest

A single tree overfits and wobbles. The fix is almost embarrassingly simple, and it is one of the most powerful ideas in machine learning.

In Lesson 1 you saw one deep tree hit 96% on training data but only 66% on test, and shift wildly when the data changed. Now you will turn that weakness into a strength.

  • Watch averaging crush a tree's variance, live
  • See why bootstrap samples make trees differ
  • Meet the one trick that makes a forest beat plain bagging

Prerequisites: Lesson 1 (decision trees, and why one deep tree overfits).

The idea

The wisdom of crowds

Ask one noisy expert and you get a noisy answer. Ask hundreds and average them, and the random errors cancel while the real signal survives. That is the whole trick behind a forest.

Key Insight
If each tree has variance \(\sigma^2\) and the trees are independent, the average of \(B\) of them has variance \(\sigma^2 / B\). More trees, steadier predictions: the random errors average away.

The catch is hiding in one word: independent. Hold that thought.

The key experiment

Build a forest, tree by tree

Each tree below is grown deep (the jagged, overfitting kind from Lesson 1) on its own random resample. Drag the slider to average more of them, and watch the boundary smooth out while accuracy climbs.

No single tree got better. You did not prune or tune them. The average simply cancels their individual mistakes.

Check yourself

Where did the accuracy come from?

As you dragged the slider and averaged more trees, the boundary smoothed and accuracy climbed. What actually got better?

Right. Each tree stays noisy, but their mistakes point in different directions, so the average lands closer to the truth.
Nothing was pruned. Every tree stays deep and overfit; averaging is the only thing at work.
The catch

Clones do not help

Averaging only cancels errors if the trees make different mistakes. Grow a hundred identical trees and their average is just that same tree again. No variance reduction at all.

So the real engineering problem of a random forest is: how do we force the trees to be different on purpose? There are two tricks, and a forest uses both.

Trick 1

Give each tree different data

Before growing a tree, draw a bootstrap sample: pick rows from the training set at random with replacement, until you have a set the same size. Some rows appear twice, others not at all.

Note
On average each bootstrap leaves out about 37% of the rows. A tree never sees its left-out rows, so they make a built-in test set for that tree. Those are the out-of-bag rows, and they power the free OOB error you will meet in Lesson 3.

Different data in means a different tree out. But there is a sneakier source of sameness left to kill.

Trick 2, the motivation

Bootstrap is not enough

Suppose one feature, say tenure, is much more predictive than the rest. Then every tree, even on its own bootstrap sample, will choose tenure for its very first split. The trees end up looking nearly identical near the top.

Statisticians call this correlation between the trees, and it is poison for averaging: remember the variance only divides by B when the trees are independent. Correlated trees barely reduce variance.

With pairwise correlation \(\rho\), the averaged variance is \(\rho\sigma^2 + \frac{1-\rho}{B}\sigma^2\). The second term shrinks as you add trees, but the first does not: \(\rho\sigma^2\) is a floor you cannot average away. The only way down is to lower \(\rho\), which is exactly what the next trick does.

The trick that names it

Random features

Here is the fix that makes it a random forest: at each split, the tree may only consider a random handful of features, not all of them. The dominant feature cannot win every time, so the trees diverge. Toggle below and watch what each tree picks for its first split.

Check yourself

Quick question

Why does a random forest let each split consider only a random subset of the features?

Exactly. Restricting features stops the dominant one from winning every split, so the trees disagree more, and uncorrelated errors cancel better when averaged.
Each tree actually gets slightly weaker. The point is the forest, not the tree.
Putting it together

The whole forest, in three rules

That is it. A random forest is just these three ideas stacked:

  1. Bootstrap: grow each tree on its own random resample of the rows.
  2. Random features: at each split, only consider a random subset (mtry) of the features.
  3. Average: let all the trees vote; the majority (or mean) is the forest's answer.
Key Insight
Each tree is a low-bias, high-variance learner. Bootstrap and random features make them diverse; averaging cancels the variance. You keep the low bias and throw away most of the variance, with almost no tuning.
Your turn

Grow the forest in R

First, load the randomForest package and build the churn data to grow the forest on. Each lesson runs in a fresh R session, so we set it all up right here (run this once):

That averages 200 decorrelated trees: bootstrap, random features and voting, all in one call.Set ntree = 200 to average 200 trees.
Show answer
rf <- randomForest(churned ~ ., data = train,
                   ntree = 200, mtry = 3)
Go deeper

References

Lesson 2 complete

You now understand why a random forest works, not just what it does. Next, you will build and tune a real one.

Up next, Lesson 3: Training, tuning and reading a forest in R. OOB error, tuning mtry and trees on a live model, reading variable importance, and the code to do it for real, ending in your Machine Learning certificate.