Chapter 105 – Decision Trees Part 2: How Models Choose the Best Data Splits
1. Introduction: Translating Theory into Python Code
Previous topics covered Gini Impurity, Entropy, and Information Gain. This module implements a Decision Tree classifier in Scikit-Learn, inspects how features are ranked for splits, exports human-readable rules, and reads feature importances.
2. Environment Setup and Library Imports
- numpy — array operations
- DecisionTreeClassifier — core classifier from sklearn.tree
- export_text — plain-text tree structure
- make_classification — synthetic classification data
import numpy as np
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.datasets import make_classification
print("Libraries loaded successfully!")
3. Synthetic Dataset Generation and Inspection
A small controlled set keeps the logic easy to visualize: 12 samples, 2 informative features, no redundant features, random_state=42 for reproducibility.
X, y = make_classification(
n_samples=12,
n_features=2,
n_informative=2,
n_redundant=0,
random_state=42
)
for i in range(8):
print(f"Sample {i+1}: Feature 1 = {X[i, 0]:.2f}, Feature 2 = {X[i, 1]:.2f} -> Class = {y[i]}")
Among the first 8 records, class mix is typically a few Class 0 and more Class 1 examples.
4. Model Training and Hyperparameter Tuning
- max_depth=2 — limits depth to reduce overfitting
- criterion='gini' — Gini Impurity for split quality (or 'entropy' for Information Gain)
- random_state=42 — reproducible splits
tree = DecisionTreeClassifier(
max_depth=2,
criterion='gini',
random_state=42
)
tree.fit(X, y)
print("Model training complete!")
5. Extracting Decision Rules & Explainable AI (XAI)
tree_rules = export_text(tree, feature_names=['feature_1', 'feature_2'])
print(tree_rules)
Typical structure (example):
|--- feature_2 <= -0.19
| |--- class: 0
|--- feature_2 > -0.19
| |--- feature_1 <= 0.78
| | |--- class: 1
| |--- feature_1 > 0.78
| | |--- class: 1
Root tests feature_2. If ≤ −0.19 → Class 0; otherwise a second test on feature_1 leads to Class 1 leaves. Every threshold and branch is visible, so Decision Trees are a clear form of Explainable AI (XAI).
6. Feature Importance: Why Feature 2 Was Chosen First
importances = tree.feature_importances_
print(f"Feature 1 Importance: {importances[0]:.3f} ({importances[0]*100:.1f}%)")
print(f"Feature 2 Importance: {importances[1]:.3f} ({importances[1]*100:.1f}%)")
Example scores: Feature 1 ≈ 0.082 (8.2%), Feature 2 ≈ 0.918 (91.8%). The higher importance feature is selected for the root split because it yields the largest Gini impurity reduction.
7. Gini Impurity vs Information Gain
| Evaluation Metric |
Gini Impurity (criterion='gini') |
Information Gain (criterion='entropy') |
| Primary Goal |
Minimize class impurity / mixing in child nodes |
Maximize reduction in entropy (uncertainty) |
| Optimization Direction |
Drive value toward 0 |
Maximize gain |
| Scikit-Learn Setting |
Default: criterion='gini' |
criterion='entropy' |
| Computational Speed |
Slightly faster (no logs) |
Slightly higher cost (log operations) |
