Python Coding Classes for Children in Brno

Chapter 118 – SVM in High Dimensions

1. Curse of Dimensionality

Text, genomics, and similar domains often have hundreds or thousands of features. In high dimensions: distances lose meaning, overfitting risk rises, and kernel costs grow. SVM still works well when support vectors and regularization keep the boundary stable.

2. Support Vectors + Regularization (C)

• Support vectors anchor the hyperplane even when many features exist.
• Small C → wider margin, some training errors allowed → simpler boundary, better generalization.
• Large C → few errors allowed → boundary can twist around points → higher overfitting risk.

3. PCA vs SelectKBest

Approach	Idea
PCA	Transform features into fewer uncorrelated components (keep variance)
SelectKBest	Keep only the top-k features by a score (e.g. f_classif)

4. Four-Way Experiment (100 Features → 20)

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.decomposition import PCA
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.metrics import accuracy_score

X, y = make_classification(
    n_samples=1000, n_features=100, n_informative=10,
    n_redundant=40, n_repeated=10, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Baseline C=10 vs regularized C=1; then PCA(20) and SelectKBest(k=20)

Typical test accuracies: Baseline C=10 ~97% (overfit risk), Regularized C=1 ~95.5%, PCA 20-comp ~95.5%, SelectKBest 20-feat ~93.5%. Regularization + reducing dimensions keeps accuracy high with a simpler model.

5. Practical Takeaway

In high dimensions, prefer moderate C, and consider PCA or feature selection before RBF SVM so you cut noise and cost without giving up much accuracy.