Bagging vs boosting (random forest vs gradient boosting)

One trains models in parallel on resampled data; the other trains them one after another.

Model answer. Both combine many models (usually trees) into a stronger ensemble.

Bagging (bootstrap aggregating)

• Draw many bootstrap samples — random samples of the training data with replacement, each the same size as the original.
• Train one model independently on each sample (can run in parallel).
• Combine predictions by majority vote (classification) or averaging (regression).
• Mainly reduces variance: individual deep trees overfit in different ways, and averaging cancels much of that noise.

Random forest = bagging + feature randomness

• At each split, a tree considers only a random subset of features (commonly about √(number of features) for classification).
• This decorrelates the trees so averaging works better.
• Each bootstrap sample misses about 37% of rows ((1 − 1/n)^n ≈ 1/e ≈ 0.368); these out-of-bag rows give a free validation estimate.

Boosting

• Train models sequentially; each new model focuses on the mistakes of the ensemble so far.
• AdaBoost: up-weights misclassified examples for the next weak learner.
• Gradient boosting: each new tree is fitted to the negative gradient of the loss with respect to the current predictions (for squared error, simply the residuals), and its output is added with a small learning rate (shrinkage).
• Mainly reduces bias: many shallow, weak trees combine into a strong model.
• Popular libraries: XGBoost, LightGBM, CatBoost.

Practical trade-offs

Common follow-up: "Why use shallow trees in boosting but deep trees in bagging?" Bagging needs low-bias, high-variance base learners (deep trees) whose variance it averages away; boosting needs high-bias, low-variance weak learners (shallow trees) whose bias it reduces step by step.