Online Robotics Classes for Kids in Tallinn

Chapter 122 – Improving KNN Performance

1. Three Optimization Levers

1. Choose k wisely — bias–variance trade-off
2. Scale features — so distance is not dominated by large-magnitude columns
3. Reduce dimensions (PCA) — fewer features → faster distance checks

2. Tuning k

• Small k (e.g. 1–2): high variance, noise-sensitive, risk of overfitting.
• Large k: smoother boundary, higher bias, risk of underfitting, more compute.
• Rule of thumb: start near √n, then search with validation or a simple loop over k = 1…20.

3. Feature Scaling

Unscaled income (0–100,000) will swamp age (18–65) in Euclidean distance. Fit StandardScaler on train only, then transform train and test so every feature has mean 0 and std 1.

4. PCA for Speed

With many features, each query costs O(n · d). PCA keeps high-variance components and drops noise. Example: 30 → 10 components can keep ~95% variance and the same accuracy while cutting distance work.

5. Breast Cancer Pipeline (Sketch)

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    cancer.data, cancer.target, test_size=0.25, random_state=42)

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)

best_k, best_acc = 1, 0
for k in range(1, 21):
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train_s, y_train)
    acc = accuracy_score(y_test, knn.predict(X_test_s))
    if acc > best_acc:
        best_k, best_acc = k, acc

pca = PCA(n_components=10)
X_train_p = pca.fit_transform(X_train_s)
X_test_p = pca.transform(X_test_s)
knn = KNeighborsClassifier(n_neighbors=best_k)
knn.fit(X_train_p, y_train)
print(best_k, best_acc, accuracy_score(y_test, knn.predict(X_test_p)),
      pca.explained_variance_ratio_.sum())

Typical outcome: k=5 peaks near ~97.9%; 10 PCA components keep ~95.4% variance and the same accuracy with fewer features.

6. Summary

Lever	Tool	Role
k tuning	Grid / loop 1–20	Balance bias and variance
Scaling	StandardScaler	Equal feature influence
PCA	n_components=10	Faster distances, same accuracy