LightGBM and CatBoost in R

In Lesson 1 you built a booster by hand: start at the average, fit each shallow tree to the leftover errors, and add a shrunken slice. It works. But that hand-rolled loop is slow, because at every split it sorts each feature and tries every possible threshold.

Sam's riverside kiosk has grown into a citywide bike-share, with a log of a few million rides. Each ride has a temperature, an hour, one of 300 start stations, and a weekday. To predict demand, the hand-rolled loop would have to test millions of thresholds at every split, and choke on those 300 stations. The production boosters keep the exact same boosting idea from Lesson 1 and make it fast enough for data like this.

By the end of this lesson you will be able to:

• Explain histogram splits: why bucketing a feature makes the split search fast, and why that speed holds as the data grows
• Contrast leaf-wise (best-first) tree growth with the older level-by-level way, and the knob that keeps it honest
• Turn a 300-value category into one useful number instead of 300 one-hot columns, without leaking the target
• Pick between LightGBM, CatBoost, and XGBoost for a given problem, and write the R that trains each

Prerequisites: Lesson 1, Gradient Boosting from Scratch (the residual loop and the learning rate), and you have met overfitting (the Bias-Variance Tradeoff lesson).

The slow part is the split search

Remember what a tree does at every node: it tries to find the best place to split. For a numeric feature like temperature, "best" means testing a threshold between each pair of neighbouring values, "below 8.2 vs above", "below 8.4 vs above", and so on, scoring each one. With \(n\) rows a feature can have up to \(n\) distinct values, so up to \(n\) candidate thresholds to score, per feature, at every single node.

Write \(n\) for the number of rows and \(p\) for the number of features. Scanning every candidate at a node costs on the order of \(n \cdot p\) work, and a booster grows hundreds of trees, each with many nodes. On Sam's few-million-row log that is the wall you hit: the arithmetic is simple, but there is an enormous amount of it.

Sam's citywide ride log

Every lesson starts a fresh R session, so we build a 5,000-ride slice of Sam's log right here. Each row is one ride: the temperature, the hour, the start station (one of 300), the weekday, and how many bikes went out.

Two things about this data will drive the whole lesson: temp is continuous with thousands of distinct values (the speed problem), and station has 300 levels (the categorical problem). We take them in that order.

Histogram-based splits

Here is the idea that makes LightGBM fast. Before growing any tree, it bins each numeric feature once: it lays a grid of about 255 buckets over the feature's range and files every value into a bucket. That is exactly what a histogram does, and you can see it below: press Run to draw the distribution of temp as buckets.

Now the payoff: when the tree looks for a split, it no longer tests a threshold between every pair of values. It only tests the bucket edges, about 255 of them, no matter how many rows there are. The feature was binned once, up front; every split after that is cheap.

Count the candidates

Let us count, on the real data, how many split points the two approaches consider for temp.

At just 5,000 rows the histogram already tests about 9 times fewer split points. And this is the part that matters: on Sam's full multi-million-row log the exact count keeps climbing with the data, while the histogram count stays pinned at 255. The split search stops scaling with the number of rows. The cost is a tiny loss of precision (a split can only land on a bucket edge, not between two exact values), and with 255 buckets that loss is almost always negligible.

Why histogram splits are fast

LightGBM buckets each numeric feature into a fixed number of bins (about 255) before it ever looks for a split. Why does that make training so much faster on a large dataset?

Leaf-wise growth

The second speed-and-accuracy idea is about how the tree grows. An older booster grows level by level: it splits every leaf on the current level before going deeper, so the tree stays balanced. LightGBM grows leaf-wise instead: at each step it looks at every open leaf, finds the single one whose best split would reduce the loss the most, and splits only that leaf, wherever it happens to sit.

Because it always spends its next split where it helps most, a leaf-wise tree squeezes more accuracy out of the same number of splits.

The catch with leaf-wise growth

LightGBM grows its trees leaf-wise: each step splits the single leaf that promises the biggest gain, wherever it sits. Compared with growing level by level, what must you guard against?

Why one-hot explodes

Now the second half of the story: those 300 stations. The classic way to feed a text category to a model is one-hot encoding, turning one column into one 0/1 column per level. With 300 stations that is 300 new columns. Let us see it.

One column became 300, almost all zeros. Worse than the size: a tree can now only ask "is this ride at station 154, yes or no" one station at a time, so it takes many shallow splits to say anything useful about stations, and each split sees very little data. High-cardinality categories are exactly where one-hot hurts most.

Target statistics: one number, not 300 columns

The modern boosters do not one-hot. They replace each category with a single informative number derived from the target. The simplest version is the mean target for that category: replace each station with the average demand seen at that station.

Formally, for a category value \(k\), the smoothed target statistic is

\[ \hat{x}_{k} = \frac{\sum_{j=1}^{n} \mathbf{1}[c_j = k]\, y_j \;+\; a\,p}{\sum_{j=1}^{n} \mathbf{1}[c_j = k] \;+\; a} \]

where \(c_j\) is the category of row \(j\), \(y_j\) is its target (bikes rented), \(\mathbf{1}[c_j = k]\) is 1 when row \(j\) belongs to category \(k\) and 0 otherwise, \(p\) is the global average target (the prior), and \(a\) is a small smoothing weight. In words: a station with many rides is trusted to its own mean, while a station with only a handful of rides is pulled toward the global average \(p\), so a rare station cannot swing the model on thin evidence. Here is the plain (unsmoothed) mean version in R:

One numeric column now carries what 300 one-hot columns did, and a tree can split it with a single threshold. LightGBM does a version of this internally (it sorts the categories by this statistic and splits the sorted order); CatBoost builds it in as its headline feature. But the naive mean above hides a trap.

Target leakage

Look again at how station_te was computed: each row's encoding is the mean demand of its station, and that mean includes the row's own rented value. The feature has peeked at the answer. For a busy station with hundreds of rides, one row barely moves the mean, so the peek is harmless. But for a station with only a couple of rides, the encoding is basically that row's own label handed back to the model, a feature that looks brilliant in training and collapses on new data. This is target leakage, and it is worst exactly where you most need help: rare categories.

The fix is to compute each row's encoding without that row's own target. The simplest form is leave-one-out: use the mean of the other rows at the station. CatBoost goes further with ordered target statistics: it puts the rows in a random order and, for row \(i\), uses only the rows \(j\) that come before it (\(j < i\)), so no row ever sees its own label and no later row leaks backward either. You will build the leave-one-out version next.

Leave one out

Below is a leakage-safe target encoding for station. It removes each row's own demand from the group total before dividing, so no row sees its own label, and it blends in the global average (the prior) so a rare station is not trusted blindly. Fill in the blank so the row's own rented value is the thing removed from the numerator.

prior  <- mean(rides$rented)
n_k    <- ave(rides$rented, rides$station, FUN = length)
sum_k  <- ave(rides$rented, rides$station, FUN = sum)

loo_te <- (sum_k - rides$rented + prior) / n_k
head(round(loo_te, 1))
#> [1] 63.8 62.6 62.6 68.0 66.2 69.0

LightGBM vs CatBoost vs XGBoost

Three boosters dominate tabular machine learning, and they share the histogram-split idea but differ where it counts. Here is how to choose.

There is no universal winner. LightGBM is the speed champion on very large data and tunes to top accuracy, but it needs a firmer hand on num_leaves and you must tell it which columns are categorical. CatBoost handles categoricals for you with the leakage-safe ordered statistics and tends to be excellent out of the box, so it is the low-effort pick when categories dominate. XGBoost is the dependable, widely supported default. A common workflow is to start with the one that fits your data shape, then try the others if you need the last bit of accuracy.

Which booster fits?

You have a churn dataset that is mostly high-cardinality text columns (city, device, plan, referrer), it is not huge, and you want a strong model with as little tuning as possible. Which booster is the natural first pick?

Which features mattered

After training, a booster can rank its features by how much they improved the loss across all splits, gain-based importance. For Sam's model it is no surprise that temperature and rush-hour dominate, with the station and weekday adding smaller lifts. This ranking is the first thing to look at after a model trains.

Train each for real

The two packages compile native C++ under the hood, so install them and run these blocks in your own R (they are shown here for reference and are not part of the runnable session above). The point is to see how little code it takes, and how each treats the categorical columns.

LightGBM: you hand it a numeric matrix and name which columns are categorical, so it never one-hots them.

library(lightgbm)

feat <- c("temp", "hour", "station", "weekday")
dtrain <- lgb.Dataset(
  data  = data.matrix(rides[, feat]),   # factors become integer codes
  label = rides$rented,
  categorical_feature = c("station", "weekday")   # LightGBM splits these natively
)

params <- list(objective = "regression", learning_rate = 0.05, num_leaves = 63)
fit  <- lgb.train(params, dtrain, nrounds = 300)
pred <- predict(fit, data.matrix(rides[, feat]))

lgb.importance(fit)   # gain-based feature ranking

CatBoost: you pass the raw data frame with its factor columns and just point at which ones are categorical; the ordered target statistics happen inside.

library(catboost)

feat <- c("temp", "hour", "station", "weekday")
pool <- catboost.load_pool(
  data  = rides[, feat],
  label = rides$rented,
  cat_features = c(2, 3)   # station and weekday, by 0-based column index
)

fit  <- catboost.train(pool, params = list(
  loss_function = "RMSE", learning_rate = 0.05, depth = 6, iterations = 300
))
pred <- catboost.predict(fit, pool)

Same data, same boosting idea from Lesson 1, but no one-hot in sight and fast enough for Sam's full log.

References

• Ke et al. (2017), LightGBM: A Highly Efficient Gradient Boosting Decision Tree, NeurIPS - the paper behind histogram splits and the speed tricks.
• Prokhorenkova et al. (2018), CatBoost: unbiased boosting with categorical features - ordered target statistics and ordered boosting, the leakage fix you saw here.
• LightGBM documentation: Features - histogram binning, leaf-wise growth, and native categorical splits, in plain terms.
• LightGBM R package documentation - the R interface (lgb.Dataset, lgb.train) used above.
• CatBoost R package - installing and calling CatBoost from R.

Lesson 2 complete

You now know what makes LightGBM and CatBoost fast and strong, and it is the same boosting idea from Lesson 1 with two engineering fixes and one categorical trick: histogram splits (bin each feature once, so the split search stops scaling with the data), leaf-wise growth (spend each split where it helps most, capped by num_leaves), and native categorical handling via leakage-safe target statistics instead of a one-hot explosion. You also have a clear rule for choosing among LightGBM, CatBoost, and XGBoost, and the R to train each.

Next, Lesson 3: The Hyperparameters That Matter. You met num_leaves and learning_rate in passing; now you will learn the handful of knobs that actually move a booster, how they interact, and how to set them without guessing.