Ensembles — The Wisdom of Many Models

A single expert's opinion can be wrong. A panel of independent experts, even if each is individually imperfect, tends to produce better decisions in aggregate. This insight — that combining diverse, imperfect models outperforms any single model — is the foundation of ensemble learning. Ensemble methods dominate machine learning competitions and are among the most reliable approaches in production systems. Understanding why they work requires thinking carefully about what kinds of errors models make and how combination cancels those errors.

Bagging: Reducing Variance by Averaging

Bagging (Bootstrap Aggregating, introduced by Leo Breiman in 1994) addresses the high-variance problem of unstable learners like decision trees. The key idea: if a model has high variance, training it on slightly different data produces a different model. If we train many such models and average their predictions, the average is more stable than any individual model — because the variance of an average of N independent variables is 1/N of the variance of each variable. The bagging procedure: 1. Create B bootstrap samples. Each bootstrap sample is created by drawing N examples from the training set with replacement (allowing duplicates). Each sample has the same size as the original training set but typically contains about 63% unique examples, with the rest being duplicates. 2. Train one model (e.g., a decision tree) on each bootstrap sample, independently. 3. At prediction time: for classification, take a majority vote among the B models. For regression, take the mean of the B models' outputs. A concrete example: Suppose you train 3 trees on 3 bootstrap samples. A new example gets predictions: Tree1=Spam, Tree2=Spam, Tree3=Not Spam. Majority vote: Spam. The ensemble is more confident than any individual tree because two of three agree. Out-of-bag error: Each bootstrap sample leaves out roughly 37% of the training data. These out-of-bag examples can be used to estimate generalization error without a separate validation set — each example is evaluated by the models that did not train on it.

Why does averaging help precisely? Consider B models each with variance σ² and no correlation between their errors. The variance of their average is σ²/B. With B=100 trees, variance drops to 1% of a single tree's variance. In practice, tree errors are correlated (all trained on similar data), but the variance reduction is still substantial. Bias is unchanged by averaging — if every model is biased in the same direction, the average will be too. Bagging helps with variance but not bias. This is why bagging works well for low-bias, high-variance models like deep decision trees, and does not help much for already-biased models like shallow linear regression.

Boosting: Converting Weak Learners into Strong Ones

Boosting takes a fundamentally different approach: instead of training models independently in parallel, it trains models sequentially, each one focusing on the mistakes of the previous. The AdaBoost algorithm (Adaptive Boosting): 1. Assign equal weight to every training example: wᵢ = 1/N. 2. Train a weak learner (e.g., a decision tree with depth 1, called a 'stump') on the weighted training set. 3. Compute the weighted error rate of this learner. Examples it misclassified get higher weights for the next round. 4. Compute the learner's contribution weight αₜ = 0.5 × ln((1 - error) / error). A more accurate learner gets a larger αₜ. 5. Repeat for T rounds, adding each weighted learner to the ensemble. 6. Final prediction: weighted majority vote of all T learners. Gradient Boosting generalizes this idea: each new model is trained to predict the residual errors (the mistakes) of the combined ensemble so far. This is equivalent to gradient descent in the space of functions rather than parameters. XGBoost, LightGBM, and CatBoost are highly optimized gradient boosting libraries. They have won more Kaggle competitions than any other algorithm family and are workhorses of industrial ML pipelines. Key differences from bagging: - Sequential, not parallel training - Each model is specialized on the previous model's mistakes - Boosting can reduce both bias and variance - More sensitive to noisy labels (misclassified outliers get progressively up-weighted)