Chapter 142 — PCA with Scikit-Learn — Part 2: Apply and Evaluate Dimensionality Reduction
1. Introduction & End-to-End Workflow Overview
In previous modules, we explored how Principal Component Analysis (PCA) reduces feature dimensions on tabular datasets like Breast Cancer and Wine. In this lecture, we apply PCA to a computer vision domain problem: handwritten digit image processing.
┌─────────────────────────────────────────────────────────────┐
│ End-to-End Image PCA Pipeline Workflow │
├─────────────────────────────────────────────────────────────┤
│ Step 1: Image Dataset Ingestion & Feature Analysis │
│ • Load Digits Dataset (1,797 samples × 64 pixels) │
│ │
│ Step 2: Feature Standardization (`StandardScaler`) │
│ • Center pixel features around mean = 0 │
│ │
│ Step 3: Extreme Dimensionality Reduction via PCA │
│ • Compress 64 pixel features to 2 Principal Components │
│ │
│ Step 4: Transformed Shape Diagnostic │
│ • Verify reduction from (1797, 64) down to (1797, 2) │
│ │
│ Step 5: Quantitative Variance & Information Loss Evaluation │
│ • PC1 (12.0%) + PC2 (9.56%) = 21.59% Total Variance │
│ │
│ Step 6: 2D Scatter Plot Visualization & Overlap Analysis │
│ • Plot 10 digit classes across PC1 and PC2 axes │
│ │
│ Step 7: Optimal Component Strategy (Grid Search & Elbow) │
│ • Iterate component counts to find the elbow point │
└─────────────────────────────────────────────────────────────┘
Computer Vision vs. Tabular Data Context:
- Computer Vision: Image and video datasets naturally carry massive dimensionality because every single pixel location acts as an input feature column. Managing high dimensionality is a constant operational requirement in image processing.
- Data Science / Tabular Data: Tabular client data usually contains fewer columns, but when dealing with high-dimensional plants or enterprise datasets, PCA is equally effective at compressing features.
2. Dataset Ingestion & Exploratory Data Analysis
For this project, we ingest Scikit-Learn's Digits Dataset (load_digits from sklearn.datasets):
- Dataset Size & Shape: Contains 1,797 image samples across 64 continuous pixel features (representing an 8 × 8 grid of pixel values per image).
- Target Classes: Contains 10 target classes representing handwritten digits from 0 through 9.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
# 1. Load the Digits dataset
digits = load_digits()
# 2. Extract input feature matrix X (pixel grid) and target labels y
X = digits.data
y = digits.target
print(f"Dataset Shape: {X.shape}") # Output: (1797, 64)
print(f"Number of Unique Digit Classes: {len(np.unique(y))}") # Output: 10
3. Feature Standardization (StandardScaler)
Because PCA computes spatial variance directions across features, input pixel values must be standardized prior to projection. Applying StandardScaler().fit_transform(X) centers pixel features around a mean of 0.
# 3. Standardize raw pixel feature values
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
4. Extreme Dimensionality Reduction (PCA(n_components=2))
To visualize high-dimensional image data on a 2D scatter plot, we instantiate Scikit-Learn's PCA model with n_components=2, compressing the 64 pixel features down to 2 orthogonal principal components:
# 4. Instantiate PCA and transform standardized image features
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
# 5. Verify shape transformation
print(f"Original Feature Shape : {X_scaled.shape}") # Output: (1797, 64)
print(f"Transformed PCA Shape : {X_pca.shape}") # Output: (1797, 2)
┌─────────────────────────────────────────────────────────────┐
│ Shape Transformation Breakdown │
├─────────────────────────────────────────────────────────────┤
│ • Original Image Feature Matrix : 1,797 rows × 64 pixels │
│ • Transformed PCA Matrix : 1,797 rows × 2 components│
│ │
│ Orthogonality Property: │
│ PC1 and PC2 are strictly ORTHOGONAL (perpendicular), │
│ guaranteeing that both composite axes represent │
│ completely independent variance patterns. │
└─────────────────────────────────────────────────────────────┘
5. Quantitative Variance & High Information Loss Diagnostics
To evaluate how much original image information was preserved after compressing 64 pixels down to 2 components, we extract the Explained Variance Ratio (explained_variance_ratio_):
# 6. Extract individual and total explained variance ratios
ratios = pca.explained_variance_ratio_
pc1_var = ratios[0]
pc2_var = ratios[1]
total_preserved_var = np.sum(ratios)
print("--- Explained Variance Ratio Diagnostics ---")
print(f"PC1 Variance Contribution : {pc1_var * 100:.2f}%") # 12.00%
print(f"PC2 Variance Contribution : {pc2_var * 100:.2f}%") # 9.56%
print(f"Total Preserved Variance : {total_preserved_var * 100:.2f}%") # 21.59%
--- Explained Variance Ratio Diagnostics ---
PC1 Variance Contribution : 12.00%
PC2 Variance Contribution : 9.56%
Total Preserved Variance : 21.59%
┌─────────────────────────────────────────────────────────────┐
│ Variance Coverage Assessment │
├─────────────────────────────────────────────────────────────┤
│ • PC1 Explained Variance : 12.00% (0.12) │
│ • PC2 Explained Variance : 9.56% (0.0956) │
│ • Total Preserved Variance: 21.59% (0.2159) │
│ │
│ Trade-Off Explanation: │
│ Compressing 64 features to 2 components yields a high │
│ information loss (retaining only 21.59% variance). │
│ However, this extreme reduction is done intentionally to │
│ enable 2D visual scatter plotting. │
└─────────────────────────────────────────────────────────────┘
Analytical Variance Highlights:
- Low Individual Contributions: PC1 captures 12.0% (0.12) of total data variance, and PC2 captures 9.56% (0.0956).
- Total Preserved Variance (21.59%): Summing PC1 and PC2 yields a total preserved variance of 21.59% (0.2159).
- Purpose of Extreme Reduction: While retaining only 21.59% variance incurs high information loss, reducing data to 2 components serves an exploratory purpose: allowing data scientists to plot 2D graphs and gain visual insights into client datasets.
6. 2D Data Visualization & Class Overlap Diagnostics
We plot the 1,797 image samples across the two principal components using matplotlib.pyplot:
# 7. Scatter plot PC1 vs PC2 colored by 10 digit classes
plt.figure(figsize=(9, 6))
scatter = plt.scatter(
X_pca[:, 0],
X_pca[:, 1],
c=y,
cmap='tab10',
alpha=0.7,
edgecolors='none'
)
plt.title('PCA Projection on Digits Dataset (64 Pixels -> 2 PCs)')
plt.xlabel('Principal Component 1 (PC1)')
plt.ylabel('Principal Component 2 (PC2)')
plt.colorbar(scatter, label='Digit Class (0 - 9)')
plt.grid(True, linestyle='--', alpha=0.5)
plt.show()
┌─────────────────────────────────────────────────────────────┐
│ 2D Scatter Plot Diagnostics │
├─────────────────────────────────────────────────────────────┤
│ Axis Value Ranges: │
│ • X-Axis (PC1) : Spans from -10.0 to +7.5 (Mean=0) │
│ • Y-Axis (PC2) : Spans from -7.5 to +10.0 (Mean=0) │
│ │
│ Visual Overlap Diagnostic: │
│ The plot displays significant class overlap across the │
│ 10 digit classes. This visual chaos directly reflects │
│ the low preserved variance (21.59%). │
└─────────────────────────────────────────────────────────────┘
Visual Diagnostic Takeaways:
- Mean-Centered Axis Ranges: PC1 values span from -10.0 to +7.5, and PC2 values span from -7.5 to +10.0, confirming that StandardScaler centered the variables around 0.
- Class Overlap & Chaos: The scatter plot displays mixing and overlap among the 10 digit categories. This visual chaos confirms that retaining only 21.59% of total variance drops too much structural detail to cleanly separate all 10 digit classes.
7. Optimal Component Selection: Grid Search & The Elbow Plot Strategy
To move beyond 2D visual exploration and identify the optimal number of components for production models, we use a Grid Search and Elbow Plot strategy:
┌─────────────────────────────────────────────────────────────┐
│ Grid Search & Elbow Plot Optimization │
├─────────────────────────────────────────────────────────────┤
│ 1. Loop across a candidate range of component counts │
│ (e.g., n_components from 2 to 40). │
│ │
│ 2. Compute total preserved variance for each count. │
│ │
│ 3. Plot Component Count vs. Preserved Variance (Elbow Plot)│
│ to identify the "elbow point" (inflection point). │
│ │
│ 4. Select the optimal K value at the elbow point. │
└─────────────────────────────────────────────────────────────┘
By iterating through component counts in a loop, plotting preserved variance produces an Elbow Plot. The inflection point (elbow) reveals the optimal number of principal components needed to maximize variance preservation while discarding redundant pixel features.
8. Summary Table: Topic 142 Pipeline Metrics
| Pipeline Stage |
Function / Syntax |
Operational Purpose |
Diagnostic Result on Digits Data |
| Data Ingestion |
load_digits() |
Ingests 1,797 samples across 64 pixel features. |
Dataset shape: (1797, 64). |
| Standardization |
StandardScaler() |
Centers pixel features around mean = 0. |
Mean-centered scaled matrix. |
| PCA Reduction |
PCA(n_components=2) |
Projects 64 pixels onto 2 orthogonal components. |
Transformed shape: (1797, 2). |
| Variance Ratios |
.explained_variance_ratio_ |
Measures variance ratio per component. |
PC1: 12.0%, PC2: 9.56%. |
| Total Coverage |
np.sum(ratios) |
Calculates total preserved variance. |
21.59% total variance. |
| Optimal Selection |
Grid Search & Elbow Plot |
Finds inflection point for optimal component count. |
Identifies ideal K value. |
