Gradient Boosting from Scratch

Gradient boosting wins more tabular machine-learning contests than any other method. Underneath the famous names (XGBoost, LightGBM, CatBoost) sits one plain idea you can build by hand: make a rough guess, look at what it got wrong, and add a small correction. Do that a few hundred times and a pile of weak little trees becomes a very strong model. You will build that loop yourself for Sam, who rents bikes from a kiosk by the river and wants to predict how many will go out on a given day from the temperature.

By the end of this lesson you will be able to:

• Explain boosting as sequential error-correction: each tree fixes what the last model still got wrong
• Grow a gradient-boosted model in R and watch the error fall
• Say what the learning rate does, and how boosting differs from a random forest

Prerequisites: you can run R, and you know what a decision tree is and that one deep tree overfits (the Random Forests course).

Learn from your own mistakes

In the Random Forests course you built trees in parallel: hundreds of deep trees, each on its own resample, all voting at once. Averaging cancelled their scattered errors. Boosting takes the opposite path, and it is worth pausing on the contrast.

A booster grows trees one after another. It starts with a single crude guess, measures how far off that guess is on every row, and grows a small tree whose only job is to predict those leftover errors. It adds that correction, measures the new (smaller) errors, and grows another tree for those. Each tree cleans up after the one before it.

That "study the leftover mistakes" step is the whole trick, so let us make it concrete with a real, tiny example.