Chapter 141 — PCA with Scikit-Learn — Part 1: Reduce the Number of Features
1. Introduction & End-to-End Workflow Overview
In machine learning, high-dimensional datasets containing dozens or hundreds of features present significant challenges. Working directly with high-dimensional data increases the risk of overfitting (where models memorize training noise) and degrades inference speed.
Principal Component Analysis (PCA) is an unsupervised learning algorithm that resolves high dimensionality by projecting features onto a smaller set of orthogonal principal components.
┌─────────────────────────────────────────────────────────────┐
│ End-to-End PCA Feature Reduction Workflow │
├─────────────────────────────────────────────────────────────┤
│ Step 1: Dataset Ingestion & Feature Scale Inspection │
│ • Load Breast Cancer Dataset (569 samples × 30 features)│
│ │
│ Step 2: Feature Standardization │
│ • Scale features using StandardScaler (Mean = 0) │
│ │
│ Step 3: Dimensionality Reduction via PCA │
│ • Compress 30 features to 2 Principal Components (PC1, PC2)│
│ │
│ Step 4: Shape Transformation Diagnostic │
│ • Verify reduction from (569, 30) down to (569, 2) │
│ │
│ Step 5: Explained Variance Ratio Evaluation │
│ • Evaluate PC1 (44.2%), PC2 (18.9%), Total (63.24%) │
│ │
│ Step 6: 2D Data Visualization & XAI Assessment │
│ • Plot PC1 vs. PC2 scatter plot & assess interpretability│
└─────────────────────────────────────────────────────────────┘
Core Mathematical Mechanics:
- Eigenvectors: Define the directional axes of the data projection.
- Eigenvalues: Quantify the amount of data variance preserved along each eigenvector axis.
- Component Selection Trade-Off:
- Selecting Fewer Components: Yields faster computation speeds but results in higher information loss.
- Selecting More Components: Preserves more information and variance, but increases computational load.
2. Dataset Ingestion & Feature Standardization (StandardScaler)
For this implementation, we utilize Scikit-Learn's Breast Cancer Wisconsin Dataset (load_breast_cancer):
- Dataset Shape: Contains 569 sample instances across 30 continuous physical features.
- High-Dimensional Candidate: Having 30 continuous features makes this dataset an ideal candidate for dimensionality reduction.
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
# 1. Load Breast Cancer dataset and extract input features X
cancer = load_breast_cancer()
X = cancer.data
y = cancer.target
print(f"Original Dataset Shape: {X.shape}") # Output: (569, 30)
Why Feature Standardization is Critical:
PCA is highly sensitive to unstandardized feature scales. When features exist on different numerical scales, features with larger raw magnitudes artificially dominate spatial direction calculations.
Applying StandardScaler().fit_transform(X) normalizes all 30 features to be centered around a mean of 0, ensuring equal feature weighting prior to PCA projection.
# 2. Standardize features using StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
3. Drastic Dimensionality Reduction (PCA(n_components=2))
We instantiate Scikit-Learn's PCA class, setting n_components=2 to compress the 30 input features down to 2 principal components (PC1 and PC2):
# 3. Instantiate PCA with 2 target components and fit_transform scaled data
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
# 4. Compare feature matrix shape before and after PCA
print(f"Shape Before PCA : {X_scaled.shape}") # Output: (569, 30)
print(f"Shape After PCA : {X_pca.shape}") # Output: (569, 2)
┌─────────────────────────────────────────────────────────────┐
│ Shape Transformation Breakdown │
├─────────────────────────────────────────────────────────────┤
│ • Original Feature Matrix : 569 samples × 30 columns │
│ • Transformed PCA Matrix : 569 samples × 2 columns │
│ │
│ Orthogonality Property: │
│ PC1 and PC2 are strictly ORTHOGONAL (perpendicular), │
│ guaranteeing they represent completely different │
│ patterns of variance in the dataset. │
└─────────────────────────────────────────────────────────────┘
By compressing 30 features into PC1 and PC2, the data representation drops from 30 columns to just 2 columns. Because PC1 and PC2 are orthogonal, they capture entirely independent data patterns.
4. Evaluating Explained Variance Ratio & Information Loss
To evaluate how much underlying information was retained after compressing 30 features into 2 components, we examine the Explained Variance Ratio (explained_variance_ratio_):
# 5. Extract explained variance ratios
variance_ratios = pca.explained_variance_ratio_
pc1_var = variance_ratios[0]
pc2_var = variance_ratios[1]
# 6. Sum individual ratios to calculate total preserved variance
total_preserved_var = pc1_var + pc2_var
print("--- Explained Variance Ratio Diagnostics ---")
print(f"PC1 Variance Contribution : {pc1_var * 100:.2f}%") # 44.20%
print(f"PC2 Variance Contribution : {pc2_var * 100:.2f}%") # 18.90%
print(f"Total Preserved Variance : {total_preserved_var * 100:.2f}%") # 63.24%
--- Explained Variance Ratio Diagnostics ---
PC1 Variance Contribution : 44.20%
PC2 Variance Contribution : 18.90%
Total Preserved Variance : 63.24%
┌─────────────────────────────────────────────────────────────┐
│ Variance Coverage Assessment │
├─────────────────────────────────────────────────────────────┤
│ • PC1 Explained Variance : 44.20% (0.442) │
│ • PC2 Explained Variance : 18.90% (0.189) │
│ • Total Variance Preserved: 63.24% (0.6324) │
│ │
│ Client Requirement Rule: │
│ Whether 63.24% variance coverage is acceptable depends │
│ on application requirements. If a client needs │
│ higher coverage (e.g., 85%+), increase `n_components` │
│ to preserve more variance and lower info loss. │
└─────────────────────────────────────────────────────────────┘
Analytical Variance Diagnostics:
- Component Contributions: PC1 accounts for 44.2% (0.442) of the total variance, while PC2 accounts for 18.9% (0.189).
- Total Preserved Variance (63.24%): Summing PC1 and PC2 yields 63.24% (0.6324) total preserved variance.
- Handling Client Requirements: Whether 63.24% variance coverage is sufficient depends on project and client requirements. If higher coverage is required, data scientists can increase the number of principal components (e.g., selecting 4 or 5 components) to reduce information loss.
5. 2D Data Visualization & Explainable AI (XAI) Assessment
Plotting 30-dimensional data directly is impossible. Compressing the dataset into 2 principal components allows us to visualize sample distribution across a 2D scatter plot:
# 7. Scatter plot PC1 vs PC2
plt.figure(figsize=(9, 6))
scatter = plt.scatter(
X_pca[:, 0],
X_pca[:, 1],
c=y,
cmap='plasma',
edgecolors='k',
alpha=0.8
)
plt.title('Breast Cancer Dataset Projected onto 2 Principal Components')
plt.xlabel('Principal Component 1 (PC1)')
plt.ylabel('Principal Component 2 (PC2)')
plt.colorbar(scatter, label='Target Class (0 = Malignant, 1 = Benign)')
plt.grid(True, linestyle='--', alpha=0.5)
plt.show()
┌─────────────────────────────────────────────────────────────┐
│ 2D Scatter Plot Diagnostics │
├─────────────────────────────────────────────────────────────┤
│ Axis Value Ranges: │
│ • X-Axis (PC1) : Ranges from -6.0 to ~17.0 │
│ • Y-Axis (PC2) : Ranges from -7.5 to ~13.0 │
│ │
│ Explainable AI (XAI) Limitation: │
│ PCA components combine multiple original features into │
│ composite axes, making individual physical features │
│ difficult to interpret directly. │
└─────────────────────────────────────────────────────────────┘
Visual & Interpretability Insights:
- Scatter Plot Ranges: PC1 values span from -6.0 to ~17.0, and PC2 values span from -7.5 to ~13.0. The plot shows separation between binary target classes (Malignant vs. Benign) alongside areas of class overlap.
- Explainable AI (XAI) Limitation: PCA does not belong to Explainable AI because principal components combine multiple original features into synthetic linear transformations, making direct physical interpretation challenging.
6. Complete Executable Python Script
Below is the complete, runnable Python code implementing the end-to-end PCA workflow:
# =====================================================================
# Complete PCA Implementation Script (Topic 141)
# =====================================================================
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
# 1. Load Dataset
cancer = load_breast_cancer()
X = cancer.data
y = cancer.target
# 2. Standardize Features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# 3. Apply PCA (30 features -> 2 components)
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
# 4. Print Shapes
print(f"Shape Before PCA : {X_scaled.shape}") # (569, 30)
print(f"Shape After PCA : {X_pca.shape}") # (569, 2)
# 5. Evaluate Explained Variance Ratios
ratios = pca.explained_variance_ratio_
print(f"PC1 Variance Ratio: {ratios[0]:.4f}") # 0.4427
print(f"PC2 Variance Ratio: {ratios[1]:.4f}") # 0.1897
print(f"Total Preserved : {np.sum(ratios):.4f}") # 0.6324 (63.24%)
# 6. Plot Results
plt.figure(figsize=(8, 5))
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y, cmap='plasma',
alpha=0.8, edgecolors='k')
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.title('PCA Reduction (569x30 -> 569x2)')
plt.grid(True, linestyle='--', alpha=0.5)
plt.show()
7. Summary Comparison: Topic 141 Workflow Components
| Pipeline Step |
Function / Tool |
Operational Purpose |
Output Result on Dataset |
| Dataset Ingestion |
load_breast_cancer() |
Ingests 569 samples across 30 continuous features. |
Matrix shape: (569, 30). |
| Standardization |
StandardScaler() |
Centers features around mean = 0 to prevent scaling bias. |
Mean-centered matrix X_scaled. |
| PCA Reduction |
PCA(n_components=2) |
Projects 30 features onto 2 orthogonal components. |
Transformed shape: (569, 2). |
| Variance Evaluation |
.explained_variance_ratio_ |
Measures variance captured per component. |
PC1: 44.2%, PC2: 18.9%. |
| Total Coverage |
np.sum(ratios) |
Calculates total preserved data variance. |
63.24% preserved variance. |
