Online Robotics Classes for Kids in Sofia

Chapter 106 – Decision Tree Depth and Overfitting Explained for Beginners

1. Introduction: Tree Depth vs Overfitting

Maximum depth (max_depth) is a key hyperparameter. Unrestricted depth tends to memorize training noise ("ratta") and hurt test performance. Controlling depth improves generalization.

• Shallow tree — may underfit
• Very deep tree — high train accuracy, weaker test accuracy (overfitting)
• Controlled depth — small train–test gap, better generalization

2. Experimental Setup

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

make_moons creates a non-linear two-class problem with overlapping half-moons—useful for probing depth limits.

3. Dataset and Train–Test Split

X, y = make_moons(n_samples=500, noise=0.25, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)
print(f"Training Samples: {len(X_train)} | Testing Samples: {len(X_test)}")

500 samples, 25% noise, 70/30 split (350 train / 150 test). Test data stays hidden during training.

4. Evaluating Depths 1–15

depths = list(range(1, 16))
train_accuracies = []
test_accuracies = []

for d in depths:
    clf = DecisionTreeClassifier(max_depth=d, random_state=42)
    clf.fit(X_train, y_train)
    train_accuracies.append(accuracy_score(y_train, clf.predict(X_train)))
    test_accuracies.append(accuracy_score(y_test, clf.predict(X_test)))

5. Plotting Train vs Test Accuracy

plt.figure(figsize=(10, 6))
plt.plot(depths, train_accuracies, label="Training Accuracy", marker="o", color="blue")
plt.plot(depths, test_accuracies, label="Testing Accuracy", marker="s", color="red")
plt.xlabel("Max Tree Depth (max_depth)")
plt.ylabel("Accuracy Score")
plt.title("Decision Tree Depth vs. Overfitting Analysis")
plt.ylim(0.80, 1.00)
plt.grid(True, linestyle="--", alpha=0.6)
plt.legend()
plt.show()

• Depth 1 — underfit (~0.83 train / ~0.82 test)
• Depth 2 — both curves rise (~0.925 / ~0.90)
• Depth 6 — small gap, strong generalization (optimal in this run)
• Depths 7–15 — train → ~100%, test drops; gap ~10% (overfitting)

6. Finding Optimal Depth in Code

best_depth_idx = np.argmax(test_accuracies)
best_depth = depths[best_depth_idx]
best_test_acc = test_accuracies[best_depth_idx]
print(f"Optimal Tree Depth: {best_depth}")
print(f"Peak Testing Accuracy: {best_test_acc * 100:.2f}%")

In this experiment, max_depth = 6 maximized test accuracy with a minimal train–test gap.

7. Deployment Checklist

• Track train and test accuracy together
• Watch for curve divergence (onset of overfitting)
• Stop increasing depth once lines pull apart; prefer the last depth where they stayed close