STEM Robotics Workshops for Kids in Florence

Chapter 154 — How to Train a Convolutional Neural Network (CNN)

1. Introduction & Core CNN Training Pipeline Overview

Training a Convolutional Neural Network (CNN) involves an iterative mathematical optimization cycle designed to minimize prediction error and maximize model accuracy across visual datasets. Just like standard artificial neural networks, training a CNN relies on a three-step cycle repeated continuously over training samples:

  ┌─────────────────────────────────────────────────────────────┐
  │              3-Step CNN Iterative Training Cycle            │
  ├─────────────────────────────────────────────────────────────┤
  │  Step 1: Forward Pass                                       │
  │     • Input image/batch is fed into the network             │
  │     • Network calculates predicted output class (y_hat)     │
  │                                                             │
  │  Step 2: Loss Calculation                                   │
  │     • Measures error between predicted output (y_hat) and   │
  │       actual desired target output (y)                      │
  │                                                             │
  │  Step 3: Backpropagation & Parameter Adjustments            │
  │     • Computes loss gradients using backpropagation         │
  │     • Adjusts connection weights and biases across layers   │
  └─────────────────────────────────────────────────────────────┘
    

1. Step 1: Forward Pass: The network receives an input image (or batch of images) and propagates the signal forward through convolutional, activation, and pooling layers to produce a predicted output (ŷ).
2. Step 2: Loss Calculation: A loss function measures the discrepancy or error between the network's predicted output (ŷ) and the ground-truth target label (y).
3. Step 3: Backpropagation Pass: The backpropagation algorithm computes error gradients with respect to every weight and bias in the network, adjusting them to reduce prediction error.

Epochs and Batch Processing:

• Epoch Definition: Completing one full pass of the entire training dataset through both the forward pass and backpropagation cycle constitutes 1 Epoch.
• Batch Processing: CNN training is executed batch-wise rather than image-wise. Data is partitioned into smaller sample batches (e.g., a batch size of 64 images).
• Batch Update Calculation: For a dataset of N total image samples trained with a batch size of 64 over 10 epochs, the network performs (N/64) × 10 total parameter updates.

2. Controlling Feature Map Spatial Dimensions — Padding Mechanics

When performing convolution, sliding a kernel filter across an image inherently reduces the output spatial dimensions and discards border details. Padding resolves this issue by adding extra pixels (typically with a numerical value of 0) around the outer perimeter of the input image.

  ┌─────────────────────────────────────────────────────────────┐
  │                     Purposes of Zero-Padding                │
  ├─────────────────────────────────────────────────────────────┤
  │  1. Spatial Dimension Control:                              │
  │     • Prevents feature maps from shrinking after convolution│
  │                                                             │
  │  2. Edge Information Preservation:                          │
  │     • Preserves visual details located at image borders that│
  │       would otherwise be lost during sliding operations     │
  └─────────────────────────────────────────────────────────────┘
    

Two Primary Types of Padding:

• Valid Padding (No Padding): No extra padding pixels are added (P = 0). As a result, the spatial size of the feature map shrinks after every convolution operation.
• Same Padding: Zero-padding pixels are added to the image borders such that the output feature map spatial size exactly equals the input image spatial size (O = W).

  ┌─────────────────────────────────────────────────────────────┐
  │            Padding Numerical Walkthrough Example            │
  ├─────────────────────────────────────────────────────────────┤
  │  Setup: Input Image = 5×5 pixels, Kernel Filter = 3×3 pixels│
  │                                                             │
  │  Case A: Valid Padding (P = 0)                              │
  │     • Output Dimension = 5 - 3 + 1 = 3                        │
  │     • Feature Map shrinks from 5×5 down to 3×3 pixels       │
  │                                                             │
  │  Case B: Same Padding (1-Cell Zero Border, P = 1)           │
  │     • One row/column of 0s added to all 4 sides             │
  │     • Formula: 5 + 2(1) - 3 + 1 = 5                         │
  │     • Feature Map size is preserved at 5×5 pixels           │
  └─────────────────────────────────────────────────────────────┘
    

• Valid Padding Example: Feeding a 5 × 5 pixel image through a 3 × 3 kernel without padding yields a reduced 3 × 3 pixel output (5 − 3 + 1 = 3).
• Same Padding Example: Adding a 1-cell border of zeros (P = 1) around the 5 × 5 image expands its effective grid to 7 × 7. Applying the 3 × 3 kernel yields an output dimension of 5 × 5 pixels (5 + 2(1) − 3 + 1 = 5), successfully preserving original spatial dimensions.

3. Controlling Downsampling & Efficiency — Stride Mechanics

Stride refers to the step size or jump distance (measured in pixels) taken by the sliding kernel window as it traverses across the image.

  ┌─────────────────────────────────────────────────────────────┐
  │                 Impact of Stride Settings                   │
  ├───────────────────────────────┬─────────────────────────────┤
  │ Small Stride (Stride = 1)     │ Large Stride (Stride = 2+)  │
  ├───────────────────────────────┼─────────────────────────────┤
  │ • Moves 1 pixel per step      │ • Jumps multiple pixels     │
  │   (Default / Minimum jump)    │   per step (Aggressive)     │
  │ • Produces LARGER feature map │ • Produces SMALLER feature  │
  │ • Captures fine image details │   map                       │
  │ • Requires HIGHER compute time│ • Dramatically REDUCES      │
  │                               │   computation time          │
  │                               │ • Risk: May discard fine    │
  │                               │   spatial information       │
  └───────────────────────────────┴─────────────────────────────┘
    

• Stride = 1 (Default): The kernel advances by 1 pixel per step, taking the finest possible sampling steps across the image.
• Stride = 2 (Aggressive Downsampling): The kernel advances by 2 pixels per step, skipping pixel locations to downsample the image.
• Computational Trade-Off: Larger strides reduce output matrix dimensions, performing downsampling that significantly decreases computation time. However, setting the stride too large risks losing essential fine visual details.

  ┌─────────────────────────────────────────────────────────────┐
  │            Stride Numerical Walkthrough Example             │
  ├─────────────────────────────────────────────────────────────┤
  │  Setup: Input Image = 7×7 pixels, Kernel Filter = 3×3 pixels│
  │                                                             │
  │  Case A: Default Stride (S = 1)                             │
  │     • Formula: (7 - 3) / 1 + 1 = 5                          │
  │     • Output Feature Map = 5×5 pixels                       │
  │                                                             │
  │  Case B: Larger Stride (S = 2)                              │
  │     • Formula: (7 - 3) / 2 + 1 = 3                          │
  │     • Output Feature Map = 3×3 pixels                       │
  └─────────────────────────────────────────────────────────────┘
    

• Stride 1 Result: Processing a 7 × 7 pixel input with a 3 × 3 kernel at S = 1 produces a 5 × 5 pixel feature map ((7 − 3)/1 + 1 = 5).
• Stride 2 Result: Processing the same 7 × 7 pixel input at S = 2 produces a smaller 3 × 3 pixel feature map ((7 − 3)/2 + 1 = 3).

4. Universal Spatial Output Dimension Formula

To calculate the spatial output dimension (Width/Height O) of a feature map for any combination of input size (W), kernel size (K), padding (P), and stride (S), use the universal spatial output equation:

O = ⌊(W − K + 2P) / S⌋ + 1

Where:

• W = Input image spatial width / height
• K = Kernel / filter spatial dimension
• P = Padding size (number of zero-layers added to each border)
• S = Stride step size
• O = Output feature map spatial width / height

5. Model Evaluation, Testing, & Preventing Overfitting

Once CNN training completes and the network achieves its target accuracy, it must be evaluated on separate data subsets.

  ┌─────────────────────────────────────────────────────────────┐
  │               Data Partitioning & Evaluation                │
  ├─────────────────────────────────────────────────────────────┤
  │  Data Partitioning Strategy:                                │
  │    • Split dataset into a larger Training Set and smaller   │
  │      Validation & Testing Sets (e.g., 70% Train / 30% Test) │
  │                                                             │
  │  Classification Evaluation Metrics:                         │
  │    • Accuracy, Loss, Precision, and Recall                  │
  │                                                             │
  │  CRITICAL TESTING RULE (Preventing Overfitting):            │
  │    • FREEZE WEIGHTS DURING TESTING                          │
  │    • Connection weights must NEVER be updated on test data  │
  │      to prevent overfitting and ensure an unbiased measure  │
  │      of model generalization                                │
  └─────────────────────────────────────────────────────────────┘
    

1. Dataset Partitioning: The dataset is divided into a larger Training Set and smaller Validation and Testing Sets (such as a 70% Training / 30% Testing split).
2. Evaluation Metrics: Model performance on classification tasks is evaluated using Accuracy, Loss, Precision, and Recall.
3. Golden Rule of Testing (Freezing Weights): During testing on unseen data, connection weights must NOT be updated. Freezing weights during testing prevents model overfitting and ensures an unbiased evaluation of how well the CNN generalizes to new data.

6. Summary Comparison Table: CNN Training Parameters

Parameter / Concept	Definition & Operational Function	Key Mathematical Effect / Rule	Primary Advantage & Trade-Off
Epoch	One complete pass across the entire training dataset.	Continuous loop until loss is minimized.	Ensures network repeatedly sees data patterns.
Batch Size	Grouping images into mini-batches for training.	Total updates = (N / Batch Size) × Epochs.	Enables efficient GPU tensor updates.
Valid Padding	No zero-padding added around image borders (P = 0).	Output size shrinks: O = W − K + 1.	Simple, but discards edge pixel details.
Same Padding	Adds zero-padding borders so output size equals input size.	Output size preserved: O = W.	Preserves edge information & image dimensions.
Stride (S)	Pixel step size taken by the sliding kernel window.	Output size formula: O = (W − K + 2P) / S + 1.	Larger stride reduces compute time but can drop fine details.
Weight Freezing	Disabling weight updates during testing on unseen data.	Weights remain static during evaluation.	Essential for preventing overfitting & testing generalization.