Ensemble Learning: Theory, Mathematics, and Practical Systems Notes¶

1. Ensemble Learning: Formal Definition¶

Ensemble learning combines multiple base learners to produce a stronger predictor than individual models.

Given base predictors \(f_1, f_2, \dots, f_M\), an ensemble predictor is:

\[ \hat f(x) = \mathcal{A}\left(f_1(x), f_2(x), \dots, f_M(x)\right) \]

where \(\mathcal{A}(\cdot)\) is an aggregation rule (average, majority vote, weighted sum, or meta-model).

Core gains: - Reduced variance (mainly bagging) - Reduced bias (mainly boosting) - Better robustness and generalization (all ensemble families)

2. Why Ensembles Work: Bias-Variance Lens¶

For regression with squared loss:

\[ \mathbb{E}[(y-\hat f(x))^2] = \text{Bias}^2[\hat f(x)] + \text{Var}[\hat f(x)] + \sigma^2 \]

Bagging reduces \(\text{Var}[\hat f]\).
Boosting reduces \(\text{Bias}[\hat f]\) by stage-wise correction.
Stacking learns a data-driven combination to reduce both in practice.

For simple averaging of equally distributed base models with pairwise correlation \(\rho\):

\[ \text{Var}(\hat f_{ens}) \approx \sigma_f^2\left(\rho + \frac{1-\rho}{M}\right) \]

Interpretation: - If models are highly correlated (\(\rho\to1\)), averaging helps little. - If models are diverse (low \(\rho\)), variance drops strongly.

Diversity is not optional; it is the engine of ensemble performance.

3. Real-World Intuition (Professor Analogy)¶

Imagine diagnosing a patient with a panel: - Doctor A focuses on symptoms, - Doctor B focuses on blood markers, - Doctor C focuses on imaging.

A combined diagnosis is often more reliable than any one doctor.

This is ensemble learning in ML: - Different learners capture different patterns, - Aggregation reduces individual blind spots.

4. Three Core Ensemble Families¶

4.1 Bagging (Bootstrap Aggregating)¶

Train many models in parallel on bootstrap samples.

Pipeline: 1. Sample training set with replacement \(M\) times. 2. Train one base learner per sample. 3. Aggregate predictions.

For classification:

\[ \hat y = \operatorname{mode}\{h_1(x),\dots,h_M(x)\} \]

For regression:

\[ \hat y = \frac{1}{M}\sum_{m=1}^{M} h_m(x) \]

Primary impact: variance reduction.

4.2 Boosting¶

Train learners sequentially; each learner focuses on mistakes made so far.

Generic additive form:

\[ F_T(x)=\sum_{t=1}^{T}\alpha_t h_t(x) \]

Primary impact: bias reduction (with risk of overfitting if overtrained/noisy labels).

4.3 Stacking¶

Train diverse base models, then train a meta-model over their predictions.

If base predictions are \(z_k(x)=f_k(x)\), then meta-model outputs:

\[ \hat y = g(z_1(x), z_2(x), \dots, z_K(x)) \]

Critical requirement: use out-of-fold (OOF) predictions to train \(g\), otherwise leakage occurs.

5. Random Forest (Canonical Bagging Model)¶

Random Forest = bagged decision trees + random feature sampling.

Two randomness sources: 1. Bootstrap data per tree. 2. Random subset of features at each split.

Why both matter: - Bagging reduces variance, - Feature subsampling decorrelates trees, amplifying variance reduction.

5.1 Practical Properties¶

Strong baseline for tabular data.
Nonlinear interaction modeling with low preprocessing burden.
OOB estimate offers near-free validation signal.
Feature importance from impurity decrease/permutation methods.

5.2 OOB Estimation¶

Each tree leaves out about 36.8% of samples (not drawn in bootstrap). Predicting those with the tree gives out-of-bag estimates.

Use OOB score to tune: - n_estimators - max_depth - max_features - min_samples_leaf

6. AdaBoost: Detailed Mathematical View¶

Assume binary labels \(y_i\in\{-1,+1\}\).

6.1 Initialization¶

\[ w_i^{(1)} = \frac{1}{N} \]

6.2 At Round \(t\)¶

Train weak learner \(h_t\) minimizing weighted error:

\[ \epsilon_t = \sum_{i=1}^{N} w_i^{(t)}\,\mathbf{1}[h_t(x_i)\neq y_i] \]

Model weight:

\[ \alpha_t = \frac{1}{2}\ln\left(\frac{1-\epsilon_t}{\epsilon_t}\right) \]

Update sample weights:

\[ w_i^{(t+1)} = \frac{w_i^{(t)}\exp(-\alpha_t y_i h_t(x_i))}{Z_t} \]

where \(Z_t\) normalizes weights to sum to 1.

Final classifier:

\[ H(x)=\operatorname{sign}\left(\sum_{t=1}^{T}\alpha_t h_t(x)\right) \]

Interpretation: - Misclassified points get larger weights. - Next learner is forced to focus on hard points.

7. Gradient Boosting: Function-Space Optimization View¶

Gradient boosting builds an additive model by descending loss gradient in function space.

Initialize:

\[ F_0(x)=\arg\min_c\sum_{i=1}^{N}L(y_i,c) \]

At stage \(t\): 1. Compute pseudo-residuals:

\[ r_i^{(t)}=-\left[\frac{\partial L(y_i,F(x_i))}{\partial F(x_i)}\right]_{F=F_{t-1}} \]

Fit weak learner \(h_t(x)\) to \(r_i^{(t)}\).
Update:

\[ F_t(x)=F_{t-1}(x)+\eta\,\gamma_t h_t(x) \]

where \(\eta\) is learning rate and \(\gamma_t\) is step size.

For squared loss, \(r_i^{(t)} = y_i - F_{t-1}(x_i)\) (plain residual).

8. Early Stopping in Ensembles (Important Regularization)¶

Early stopping is not an ensemble class, but it is essential in boosting.

Mechanism: - Track validation metric per boosting stage. - Stop if no improvement for patience rounds. - Restore best stage.

Effect: - Prevents late-stage overfitting, - Often better than blindly increasing number of estimators.

9. Stacking: Correct Training Protocol¶

Incorrect stacking often leaks target information.

Correct OOF procedure: 1. Split training data into \(K\) folds. 2. For each base model and fold: - Train on \(K-1\) folds, - Predict held-out fold. 3. Concatenate held-out predictions to form meta-features. 4. Train meta-model on these OOF meta-features. 5. Refit base models on full training data for final inference.

10. Comparative Table: Bagging vs Boosting vs Stacking¶

Aspect	Bagging	Boosting	Stacking
Training pattern	Parallel	Sequential	Base parallel + meta sequential
Main objective	Reduce variance	Reduce bias	Learn best model combination
Base learners	Usually same type	Usually weak learners	Usually diverse
Sensitivity to noise	Moderate	High (if not regularized)	Depends on base models
Interpretability	Medium	Medium-Low	Low-Medium
Typical examples	Random Forest	AdaBoost, Gradient Boosting, XGBoost-style families	Tree + linear + kernel blend

11. Worked Mini Examples¶

11.1 Majority Vote (Classification)¶

Predictions: \([1,1,0,1,0]\)

\[ \hat y = 1 \]

11.2 Averaging (Regression)¶

Predictions: \([12.5,13.0,11.5,12.0]\)

\[ \hat y = \frac{12.5+13.0+11.5+12.0}{4}=12.25 \]

11.3 AdaBoost Stage Weight¶

If \(\epsilon_t=0.2\):

\[ \alpha_t=\frac{1}{2}\ln\left(\frac{0.8}{0.2}\right)=\frac{1}{2}\ln(4)\approx0.693 \]

Lower error yields higher \(\alpha_t\), so better weak learners influence final decision more.

11.4 Gradient Boosting Update¶

If \(F_{t-1}(x_i)=20\), true \(y_i=26\), learner predicts \(h_t(x_i)=5\), learning rate \(\eta=0.1\):

\[ F_t(x_i)=20+0.1\times5=20.5 \]

12. Practical Use Cases¶

Fraud detection
Boosting captures rare nonlinear interaction patterns.
Credit risk
Stacking blends calibrated linear models and nonlinear tree models.
Demand forecasting
Random Forest/boosting handle complex feature interactions.
Medical triage/risk scoring
Ensembles improve stability versus single high-variance models.
Search ranking and recommendations
Stacked ensembles combine different relevance signals.

13. Failure Modes and Edge Cases¶

Highly correlated bagged models -> weak variance reduction.
Boosting on noisy labels -> memorization of noise.
Poorly tuned learning rate + too many estimators -> overfit.
Stacking without OOF -> severe leakage.
Class imbalance without weighting -> majority-class bias.
Distribution shift -> ensemble calibration degradation.

14. Production Checklist¶

Start with a single-model baseline.
Add Random Forest baseline.
Add boosting with early stopping.
Compare via cross-validation and calibration.
Use explainability tools (SHAP/permutation importance).
Validate latency/memory constraints.
Stress-test across temporal/data drift slices.

15. Exam-Ready Summary¶

Ensemble = combine weak/strong learners to improve generalization.
Bagging: parallel bootstrap + averaging/voting -> reduces variance.
Boosting: sequential error correction -> reduces bias.
Stacking: meta-model over base predictions -> combines strengths.
Random Forest is bagging + feature randomness.
AdaBoost reweights samples based on classification errors.
Gradient boosting fits residuals/negative gradients stage-wise.
Early stopping is critical in boosting regularization.
Diversity among base models is essential for ensemble gains.
OOF predictions are mandatory for leakage-free stacking.