Online Robotics Classes for Kids in Prague

Chapter 117 – The Kernel Trick Explained

1. When Linear Separation Fails

Many real datasets are not linearly separable (e.g. one class forming a ring around another). A straight line or flat hyperplane cannot split them cleanly. The kernel trick maps data into a higher-dimensional space where a linear boundary works again—without building that space explicitly.

2. Inner Products Instead of Explicit Mapping

A kernel function \(K(x_i, x_j) = \langle \Phi(x_i), \Phi(x_j) \rangle\) evaluates similarity in the feature space via inner products. That avoids computing full high-dimensional coordinates and keeps computation practical.

3. Common Kernels

Kernel	Formula	Use
Linear	\(x_i \cdot x_j\)	Already linearly separable data
Polynomial	\((x_i \cdot x_j + e)^d\)	Moderate curved boundaries
RBF (Gaussian)	\(\exp(-\gamma \|x_i - x_j\|^2)\)	Complex non-linear data (default in practice)

4. Circles Demo (Scikit-Learn)

from sklearn.datasets import make_circles
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score

X, y = make_circles(n_samples=200, factor=0.3, noise=0.05, random_state=42)

for name, model in {
    'Linear': SVC(kernel='linear'),
    'Polynomial': SVC(kernel='poly', degree=3),
    'RBF': SVC(kernel='rbf', gamma=1.5)
}.items():
    model.fit(X, y)
    print(name, accuracy_score(y, model.predict(X)) * 100)

Typical result on concentric circles: Linear ~69%, Poly ~71%, RBF ~100% — RBF forms a circular boundary that separates inner and outer classes.

5. Takeaway

Prefer RBF when boundaries are clearly non-linear; use linear when data is already separable with a line/plane. Polynomial sits in between for moderate curves.