STEM Robotics Workshops for Kids in Rijeka

Chapter 110 – How Random Forest Works — Part 1

1. Introduction & The Committee Analogy

A single decision tree is easy to visualize but can overfit and show high variance when trained on the full dataset. Random Forest trains many trees on different samples and combines their outputs for more stable, accurate predictions—like a committee of experts instead of one decision-maker.

Different trees see different sub-samples, so they prioritize different features at the root. That diversity helps neutralize feature bias across the forest.

2. Bootstrapping: Sampling with Replacement

Bootstrapping draws n samples from the main dataset with replacement: a row that is picked stays in the pool and can be picked again. The result for each tree:

• Some records appear more than once in that tree’s training subset
• Some records are never selected for that tree
• Each tree trains on a distinct sub-sample and learns a different view of the problem

3. Out-of-Bag (OOB) Samples

Records not selected for a tree’s bootstrap sample are its Out-of-Bag (OOB) samples. Example: 10 records total, 8 selected → 2 OOB. Because the tree never saw those OOB rows during training, they act as a built-in internal test set and support early checks on generalization without an extra external split.

4. Bagging (Bootstrap Aggregating)

After trees are trained on their bootstrap samples, predictions are combined:

• Classification — majority vote (mode). Example (Iris): 5 trees vote Setosa, Setosa, Virginica, Setosa, Versicolor → final class Setosa.
• Regression — mean of the continuous outputs from all trees.

Aggregation keeps bias low while reducing variance and improving stability.

5. Python Implementation with Scikit-Learn (Iris)

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

iris = load_iris()
X, y = iris.data, iris.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=42
)

rf_model = RandomForestClassifier(
    n_estimators=50,    # 50 trees
    max_depth=None,     # no depth limit
    bootstrap=True,     # sampling with replacement
    random_state=42,
    n_jobs=-1           # use all CPU cores
)

rf_model.fit(X_train, y_train)
y_pred = rf_model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"Random Forest Test Accuracy: {accuracy * 100:.1f}%")

On this setup the model reaches about 88.9% accuracy on the held-out test set.

6. Bootstrapping vs Bagging

Dimension	Bootstrapping	Bagging
Core process	Random sampling with replacement	Combining predictions from many bootstrapped trees
Stage	Dataset preparation (per tree)	Inference / final prediction
Main goal	Diversity + OOB samples	Lower variance, more stable predictions
Aggregation	N/A	Majority vote (class) / mean (regression)