Navigate
Roadmap Tutorials Exercises Tools Get certified
Lesson 3 of 6

The Hyperparameters That Matter

In Lesson 2 you met the fast production boosters and the settings they hand you: LightGBM alone exposes more than a hundred. Stare at that list and it is easy to freeze. The good news is that almost none of them decide whether your model is good. Just four do, and the rest are fine-tuning.

Sam's citywide bike-share is still our example: predict how many bikes go out on a given day from the day's temperature and whether it is a weekend. A booster keeps stacking small corrective trees, so the very first question is not "which model" but "how far do I let it go, and how big are its steps." That is what these four knobs control.

By the end of this lesson you will be able to:

  • Name the four hyperparameters that actually move a booster, and say what each one does
  • Explain why more trees is not always better, and read the U-shaped validation curve it produces
  • Set the learning rate and the number of trees together, because they trade off against each other
  • Set tree depth to match your data, and use regularization to pull an over-eager model back

Prerequisites: Lesson 1 (Gradient Boosting from Scratch: the residual loop, the learning rate as a shrunken slice, shallow trees in sequence) and Lesson 2 (LightGBM and CatBoost in R: you met num_leaves and learning_rate in passing). It also helps to have met overfitting as bias versus variance. You can run R and you know what a training/validation split is and what RMSE measures.

First, the short list

Four knobs, not a hundred

A hyperparameter is a setting you choose before training, as opposed to the tree splits and leaf values the algorithm learns from the data on its own. Boosters expose dozens of them, but they are not equally important. Change most of them and your validation error barely twitches. Four of them move it a lot, and they are the same four in every gradient booster (XGBoost, LightGBM, CatBoost), just under different names.

  1. Number of trees (how many corrective trees you add in total).
  2. Learning rate (how big a slice of each tree you keep).
  3. Tree depth (how complex each individual tree is allowed to be).
  4. Regularization (extra brakes that stop any single tree from memorizing).

The whole rest of this lesson is these four, one at a time, and then the part that trips people up: they are not independent. Turn one and you often have to compensate with another.