Online Robotics Classes for Kids in Nicosia

Chapter 112 – Feature Importance and Out-of-Bag Error

1. Two Practical Diagnostics

Beyond accuracy, Random Forest offers:

• Feature importance — how much each variable reduces impurity across all trees (useful for ranking drivers and simplifying models).
• Out-of-Bag (OOB) error — internal validation on samples left out of each tree’s bootstrap set (~1/3 of data), so you often do not need a separate hold-out split.

2. Feature Importance (3 Steps)

1. At every split that uses feature f, measure the drop in Gini/Entropy (or gain in Information Gain).
2. Sum those reductions for f over all such nodes in all trees; normalize so scores lie between 0 and 1 (or sum to 1).
3. Rank features by that score — higher = more predictive.

Conceptually: Importance(f) = sum of ΔImpurity over nodes that split on f.

Example (house prices): Size 0.55, Location 0.25, Bedrooms 0.15, Age 0.05 — Age could be dropped with little loss.

3. Out-of-Bag Error

With sampling with replacement, about 33% of rows are left out of any given tree. Those OOB rows act as a built-in test set for that tree.

1. Identify OOB samples per tree
2. Predict them with that tree
3. Aggregate predictions for each sample across trees where it was OOB, then compare to true labels → OOB accuracy and OOB error (1 − accuracy)

Example: 1000 samples → ~333 OOB per tree; if 300 of 333 are correct, OOB accuracy ≈ 90%, error ≈ 10%.

4. Python (Breast Cancer, Full Data + OOB)

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
import numpy as np

cancer = load_breast_cancer()
X, y = cancer.data, cancer.target
feature_names = cancer.feature_names

rf = RandomForestClassifier(
    n_estimators=100,
    oob_score=True,   # internal OOB evaluation
    bootstrap=True,
    random_state=42,
    n_jobs=-1
)
rf.fit(X, y)  # no train_test_split required for OOB

oob_accuracy = rf.oob_score_
oob_error = 1.0 - oob_accuracy
print(f"OOB Accuracy: {oob_accuracy * 100:.2f}%")
print(f"OOB Error:    {oob_error * 100:.2f}%")

importances = rf.feature_importances_
sorted_idx = np.argsort(importances)[::-1]
print("Top 5 features:")
for rank, i in enumerate(sorted_idx[:5], 1):
    print(f"{rank}. {feature_names[i]}: {importances[i]:.4f}")

5. Typical Output

• OOB accuracy ≈ 96.13%, OOB error ≈ 3.87%
• Top features often: worst area (~0.15), worst concave points (~0.14), mean concave points (~0.10), worst perimeter, worst radius

OOB scoring lets the forest validate itself on the full dataset without an external split — especially useful on smaller data.