Ensemble Methods in Machine Learning: Bagging, Boosting & Random Forests
Ensemble learning = combining multiple models to outperform any single one. Core insight: diverse errors cancel; shared signals amplify.
Three Pillars
1. Bagging (Bootstrap Aggregating)
- Create B bootstrap samples (random with replacement)
- Train base learner on each
- Aggregate: majority vote (classification) or average (regression)
- Reduces variance – stabilizes unstable learners (e.g., deep trees)
- Flagship: Random Forest – bagging + random feature subsets per split
2. Boosting (Sequential Correction)
- Train weak learners sequentially
- Each focuses on predecessors' errors (re-weighted data)
- Aggregate: weighted sum
- Reduces bias – builds strong learner from weak ones
- Variants: AdaBoost (adaptive weights), Gradient Boosting (fit residuals), XGBoost/LightGBM/CatBoost (optimized implementations)
3. Stacking (Meta-Learning)
- Train diverse base models
- Train meta-model on their predictions
- Captures complementary strengths
Why Ensembles Win
- Lower generalization error
- Robust to outliers/noise
- Handle complex decision boundaries
- Standard in Kaggle competitions & production ML
Trade-off: complexity, compute, interpretability. But for predictive power? Ensemble.