XGBoost Exercises in R: 20 Real-World Practice Problems

Explanation: xgb.train() is the low-level API and takes an xgb.DMatrix rather than a raw data frame, which is more efficient for repeated calls because the matrix is preprocessed once. The default eta = 0.3 is aggressive for small data, but acceptable for a baseline. The newer xgboost() 2.x wrapper accepts data frames directly, but xgb.train() remains the workhorse you see in production code and tuning loops.

Explanation: binary:logistic outputs probabilities in (0, 1); use binary:logitraw if you want pre-sigmoid scores for ensembling. The labels must be 0/1 integers, not factors. With only 32 rows the train log-loss will collapse to near zero (overfit) - that's expected for a memorization exercise. Real binary tasks need a holdout, which we build in later problems.

Explanation: multi:softprob returns a row of length num_class per observation (probabilities summing to 1), whereas multi:softmax returns only the predicted class index. Most pipelines want softprob so you can threshold, calibrate, or stack downstream. Forgetting the -1L shift to zero-indexed labels is the classic beginner trap: XGBoost will silently fail or train a degenerate model.

Explanation: XGBoost returns multiclass probabilities as a single flat vector of length nrow * num_class. byrow = TRUE is critical: XGBoost emits one row at a time, so reshaping column-major would scramble the per-observation probabilities. Alternatively, pass reshape = TRUE to predict() and skip the manual matrix step - that argument exists from xgboost 1.5 onward.

Explanation: eta (a.k.a. learning_rate) shrinks each new tree's contribution. Lowering it from 0.3 to 0.05 typically requires roughly 5-10x more rounds to reach the same training fit, but the resulting model generalizes better because each step is smaller and more cautious. In production, a common pattern is eta = 0.05 plus early_stopping_rounds = 20 so you let the model decide when to stop.

Explanation: The default max_depth = 6 allows trees to capture up to 6-way interactions, which often overfits small or noisy datasets. Depths of 3-5 with hundreds of rounds usually outperform deep trees with fewer rounds on tabular data. Setting max_depth = 0 switches to lossguide growth (LightGBM-style), where you control complexity via max_leaves instead.

Explanation: Early stopping monitors the LAST evaluation metric on the LAST element of the watchlist, so the order matters: put validation last. After fitting, always score with predict(model, newdata, iterationrange = c(1, model$best_iteration + 1)) (or rely on the default in xgboost 1.5+) so you don't accidentally use the overfit late-round trees. early_stopping_rounds is the patience parameter; 20-50 is typical.

Explanation: model$evaluation_log is a data.table (xgboost depends on data.table internally), so coercing to a plain data frame avoids surprises if you later dplyr::filter it. The log is the foundation of every learning curve plot; a divergence between train and validation RMSE that grows after some iteration is the visual signal that early stopping is doing its job.

Explanation: xgb.cv returns CV diagnostics but does NOT return a fitted booster you can call predict() on. The standard pattern is to use xgb.cv to choose nrounds (via best_iteration), then call xgb.train ONCE on the full data with that nrounds value. With 32 rows the fold size is tiny (6-7 rows), so RMSE estimates here are noisy; in production, use repeated CV via repeats = 3 or run on a larger sample.

Explanation: caret's xgbTree method wraps xgb.train and requires all seven xgbTree hyperparameters in the grid even if you only vary two - that's a common stumbling block. For more flexible search (Bayesian optimization, random search over wider ranges), the modern alternative is the tidymodels stack with tune::tune_grid or tune::tune_bayes, which integrates cleanly with parsnip::boost_tree.

Explanation: min_child_weight is the minimum sum of instance weights (basically observation count for unweighted data) required in a leaf. Higher values force the tree to keep larger groups together, which suppresses noise. On heavily noisy data like this, jumping from 1 to 20 typically buys a 5-10% RMSE reduction - small but free. It pairs naturally with gamma (minimum loss reduction to split) as the two main "stop splitting" levers.

Explanation: Row subsampling is the boosting analogue of bagging: each round trains on a different 70% slice of rows. Values of 0.5-0.9 are typical; below 0.5 you usually need more rounds. Critically, this is the only randomness source you have in stock XGBoost besides feature subsampling, so reproducibility requires set.seed() BEFORE the xgb.train call when subsample < 1.

Explanation: XGBoost offers three nested column-sampling knobs: colsample_bytree (per tree), colsample_bylevel (per depth level), and colsample_bynode (per split). They multiply: a per-tree of 0.5 and a per-level of 0.5 means each level sees 25% of columns. Per-tree is the most common; per-node mimics random forest's column choice at every split and helps when features are highly correlated.

Explanation: alpha (L1) drives some leaf weights to exactly zero, creating sparse trees, while lambda (L2, default 1) just shrinks them. On clean small data you rarely need either; on wide noisy data (text, embeddings, leaked-feature scenarios) tuning these often beats tuning depth. They penalize the LEAF SCORES, not the inputs - that's different from glmnet, where L1/L2 sits on the coefficient vector.

Explanation: Three columns matter: Gain (loss improvement, the one you usually report), Cover (number of observations affected, normalized), and Frequency (raw split count). Gain is the most loss-aware ranking; Frequency can mislead you because XGBoost may split many times on a high-cardinality column even when each split barely helps. For categorical-encoded features, the order can swap dramatically between metrics.

Explanation: SHAP values decompose a single prediction into per-feature additive contributions plus a BIAS term that equals the average training prediction. By construction, sum(ex_5_2) equals the model's prediction for that row, so predict(ex_1_1, xnew[1, , drop = FALSE]) matches. For interaction effects, pass predinteraction = TRUE instead; you get a n by p by p array. The SHAPforxgboost package builds nice plots on top of this.

Explanation: xgb.plot.importance() is a thin wrapper around base graphics; for ggplot-styled output, call it with plot = FALSE to suppress the plot and then build your own with ggplot2. The Importance column is identical to Gain by default; switch via measure = "Cover" or "Frequency". Always report at least Gain and Frequency together when communicating with non-technical stakeholders, because Frequency without Gain misleads on high-cardinality features.

Explanation: xgb.save writes the native XGBoost binary format, which is portable across R/Python/JVM/CLI - that's the file format you ship to production. saveRDS will also serialize the booster, but the resulting .rds only loads back into R and ties you to xgboost's R-side internal layout (it can break across major version bumps). For long-term reproducibility, also pin the xgboost version and the booster's nfeatures and feature names.

Explanation: dgCMatrix is the standard sparse format from the Matrix package and is what sparse.model.matrix() returns for one-hot encoded design matrices. XGBoost accepts it directly - no need to densify. On wide data (thousands of one-hot columns) the memory savings are 10-100x. The nfeatures slot is a guardrail: a mismatch between training and prediction feature counts will throw a clear error rather than silently producing nonsense.

Explanation: Monotone constraints are a regulator-friendly way to force business rules into the model: -1 means "as the feature increases, predictions must not increase", +1 enforces the opposite, 0 leaves it unconstrained. The vector length must match the feature count and order. Constraints typically cost a tiny bit of accuracy but make models defensible in credit risk, insurance pricing, and any regulated domain.