STEM Robotics Workshops for Kids in Larissa

Chapter 139 – PCA Explained: Eigenvectors and Eigenvalues Made Simple

1. Introduction & Mathematical Foundations of PCA

In unsupervised machine learning, Principal Component Analysis (PCA) is an essential algorithm used to reduce feature dimensionality while preserving the underlying structure of a dataset. While PCA simplifies high-dimensional data, its underlying mathematical power relies on two core linear algebra concepts: Eigenvectors and Eigenvalues.

  ┌─────────────────────────────────────────────────────────────┐
  │              Mathematical Foundations of PCA                │
  ├─────────────────────────────────────────────────────────────┤
  │  Input Dataset (Covariance Matrix A)                        │
  │        │                                                    │
  │        ├── Linear Algebra Deconstruction (A v = λ v)       │
  │        │                                                    │
  │        ├── 1. Eigenvectors (v): Direction of Components     │
  │        └── 2. Eigenvalues (λ): Magnitude / Variance Scale   │
  │                                                             │
  │  Output:                                                    │
  │    Orthogonal Principal Components Preserving Max Variance  │
  └─────────────────────────────────────────────────────────────┘

Understanding how eigenvectors and eigenvalues operate on data covariance matrices unlocks the exact mechanics behind how PCA compresses features without losing critical variance.

2. Core Concepts: Eigenvectors vs. Eigenvalues

A. What is an Eigenvector?

When a matrix transformation is applied to vectors in space, most vectors change both their magnitude and direction. An Eigenvector is a special invariant vector whose direction remains unchanged when a linear transformation is applied.

  ┌─────────────────────────────────────────────────────────────┐
  │                 Eigenvectors vs. Eigenvalues                │
  ├───────────────────────────────┬─────────────────────────────┤
  │ Concept                       │ Mathematical Meaning        │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Eigenvector (v)               │ Directional axis that stays │
  │                               │ invariant during transform  │
  │ Eigenvalue (λ)                │ Scalar factor quantifying   │
  │                               │ the amount of stretch/scale │
  └───────────────────────────────┴─────────────────────────────┘

Because eigenvectors maintain their directional orientation under transformation, they act as fundamental, invariant axes that capture the true structural trends of a dataset.

B. What is an Eigenvalue?

An Eigenvalue is a scalar value (number) that quantifies how much an eigenvector is stretched or compressed during a matrix transformation.

C. The Fundamental Equation

The mathematical relationship between a covariance matrix, its eigenvector, and its eigenvalue is expressed by the fundamental formula:

A v = λ v

Where:

• A = The Covariance Matrix of the standardized dataset.
• v = The Eigenvector (representing direction).
• λ = The Eigenvalue (a scalar representing stretch magnitude / variance).

3. The Step-by-Step Mathematical Workflow in PCA

The mathematical execution of PCA follows a clear 5-step sequence:

  ┌─────────────────────────────────────────────────────────────┐
  │                   PCA Mathematical Workflow                 │
  ├─────────────────────────────────────────────────────────────┤
  │  Step 1: Standardize Dataset                                │
  │     • Center data around Mean = 0 and Variance = 1          │
  │                                                             │
  │  Step 2: Compute Covariance Matrix (A)                      │
  │     • Capture pairwise covariances between features         │
  │                                                             │
  │  Step 3: Extract Eigenvalues (λ) & Eigenvectors (v)         │
  │     • Solve A v = λ v using linear algebra                  │
  │                                                             │
  │  Step 4: Sort Components by Eigenvalue Magnitude            │
  │     • Rank eigenvectors by variance (λ) descending          │
  │                                                             │
  │  Step 5: Compute Explained Variance Ratio                   │
  │     • Divide each λᵢ by Σλ                                  │
  └─────────────────────────────────────────────────────────────┘

1. Data Standardization: Standardize features so they have a mean of 0 and a variance of 1, ensuring that feature scales do not distort eigenvector calculations.
2. Covariance Matrix Computation (A): Compute the covariance matrix A to inspect pairwise feature relationships and variance spreads across all dimensions.
3. Eigen-Decomposition: Solve A v = λ v to extract eigenvectors (component directions) and eigenvalues (component variances).
4. Ranking & Component Selection: Sort eigenvectors in descending order of their corresponding eigenvalues. Eigenvectors paired with the largest eigenvalues capture the maximum variance. For instance, in a 3D dataset, selecting the top two eigenvectors with the largest eigenvalues reduces the space to 2D while retaining maximum data trend variance.
5. Explained Variance Ratio: Calculate the proportion of total variance captured by each principal component.

4. Key Mathematical Properties & Advantages

  ┌─────────────────────────────────────────────────────────────┐
  │                 Key Mathematical Properties                 │
  ├───────────────────────────────┬─────────────────────────────┤
  │ Property                      │ Operational Significance    │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Mutual Orthogonality          │ Eigenvectors sit at 90° to  │
  │                               │ each other (uncorrelated)   │
  │ Total Variance Conservation   │ Σλᵢ = Total Data Variance   │
  │ Variance Preservation         │ Reduces features with       │
  │                               │ minimal variance loss       │
  └───────────────────────────────┴─────────────────────────────┘

• Mutual Orthogonality: Eigenvectors extracted from a symmetric covariance matrix are strictly orthogonal (perpendicular) to one another. This guarantees that the resulting principal components are completely uncorrelated.
• Total Variance Conservation: The sum of all extracted eigenvalues equals the total variance present in the dataset.
• Minimal Variance Loss: By retaining eigenvectors associated with large eigenvalues and discarding those with near-zero eigenvalues, dimensionality is reduced while keeping data variance loss to a minimum.

5. Python Implementation Step-by-Step (numpy & linalg.eig)

Below is the complete step-by-step Python code demonstrating how to standardize a dataset, compute its covariance matrix, extract eigenvalues and eigenvectors using numpy.linalg.eig, and calculate explained variance ratios:

import numpy as np

# 1. Define raw sample dataset array
data = np.array([
  # sample rows omitted for brevity in notes
])

print("Raw Dataset Array:\n", data)

# 2. Step 1: Standardize dataset (Center around Mean = 0, Variance = 1)
mean = np.mean(data, axis=0)
std = np.std(data, axis=0)
data_standardized = (data - mean) / std

print("\nStandardized Data Array (Mean-Centered):\n", np.round(data_standardized, 2))

# 3. Step 2: Compute Covariance Matrix of the standardized data
# rowvar=False indicates that columns represent features
cov_matrix = np.cov(data_standardized, rowvar=False)

print("\nComputed Covariance Matrix (A):\n", np.round(cov_matrix, 2))

# 4. Step 3: Compute Eigenvalues and Eigenvectors using linear algebra module
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)

print("\nCalculated Eigenvalues (λ):\n", np.round(eigenvalues, 2))
print("\nCalculated Eigenvectors Matrix (v):\n", np.round(eigenvectors, 2))

# 5. Step 4: Sort Eigenvalues and Eigenvectors in descending order
sorted_indices = np.argsort(eigenvalues)[::-1]
sorted_eigenvalues = eigenvalues[sorted_indices]
sorted_eigenvectors = eigenvectors[:, sorted_indices]

# Assign final Principal Components from eigenvectors
pc1 = sorted_eigenvectors[:, 0]
pc2 = sorted_eigenvectors[:, 1]

print("\nPrincipal Component 1 (PC1 Vector):", np.round(pc1, 2))
print("Principal Component 2 (PC2 Vector):", np.round(pc2, 2))

# 6. Step 5: Compute Explained Variance Ratio
explained_variance_ratio = sorted_eigenvalues / np.sum(sorted_eigenvalues)

print("\n--- Explained Variance Ratio ---")
print(f"PC1 Variance Contribution: {explained_variance_ratio[0] * 100:.1f}%")
print(f"PC2 Variance Contribution: {explained_variance_ratio[1] * 100:.1f}%")

Execution Output Analysis:

• Covariance Matrix (cov_matrix): Captures feature interactions (showing maximum covariances around 2.7 and 2.5).
• Eigenvalues ([3.85, 1.34]): The eigenvalue of 3.85 indicates high vector stretching along the primary axis, while 1.34 indicates lower stretch along the secondary axis.
• Eigenvectors (PC1 & PC2): The extracted vectors ([-0.67, -0.73] and [-0.73, 0.67]) become the final Principal Components.
• Explained Variance Ratio (74.2% & 25.8%): Dividing each eigenvalue by the sum of eigenvalues reveals that PC1 captures 74.2% (0.742) of the total data variance, while PC2 captures 25.8% (0.258).

6. Summary Comparison: Eigenvectors vs. Eigenvalues

Concept	Mathematical Role	Role in PCA	Python Attribute / Formula
Covariance Matrix (A)	Summarizes pairwise feature covariances.	Input matrix used to extract components.	np.cov(data_std, rowvar=False)
Eigenvector (v)	Directional vector invariant to linear transformations.	Forms the directional axes of Principal Components.	eigenvectors from np.linalg.eig
Eigenvalue (λ)	Scalar measuring stretch magnitude along eigenvector.	Quantifies the amount of variance captured by each component.	eigenvalues from np.linalg.eig
Orthogonality	Vectors sit at 90° right angles to each other.	Guarantees components are uncorrelated.	Eigenvector dot product = 0
Explained Variance Ratio	Ratio of eigenvalue to total sum of eigenvalues.	Measures percentage of total data variance explained.	eigenvalues / np.sum(eigenvalues)