Coding and Robotics Classes for Kids in Genoa

Chapter 147 — Neural Network Training Explained Step by Step

1. Introduction & The Core Purpose of Training

In artificial neural networks, training refers to the process of teaching a model to minimize its prediction error and maximize its accuracy. A well-trained neural network is one that achieves minimal error when processing data.

  ┌─────────────────────────────────────────────────────────────┐
  │                 Biological Human Analogy                    │
  ├─────────────────────────────────────────────────────────────┤
  │  Human Learning Mechanism:                                  │
  │    • Humans learn by being repeatedly exposed to examples.  │
  │    • Extracted patterns are stored in human memory.         │
  │                                                             │
  │  Neural Network Training Analogy:                           │
  │    • Neural networks learn by repeatedly processing         │
  │      input data samples across multiple iterations.         │
  │    • Learns underlying patterns and updates weights.        │
  └─────────────────────────────────────────────────────────────┘
    

The Human Learning Analogy

Neural network training mirrors the way humans learn. When humans are trained, they are shown examples repeatedly to improve their understanding and extract underlying patterns into memory. Similarly, a neural network must be exposed to input data repeatedly to learn data patterns and improve its predictive performance.

2. The 5-Step Training Pipeline Flow

The complete training execution pipeline follows a structured 5-step sequence:

  ┌─────────────────────────────────────────────────────────────┐
  │               5-Step Neural Network Pipeline                │
  ├─────────────────────────────────────────────────────────────┤
  │  Step 1: Parameter Initialization                           │
  │     • Initialize weights (w) and biases (b) using           │
  │       random number generation.                             │
  │                                                             │
  │  Step 2: Forward Pass                                       │
  │     • Feed training data into input layer ──► Moves toward  │
  │       output layer to calculate predictions.                │
  │                                                             │
  │  Step 3: Loss Calculation                                   │
  │     • Compare predicted values (ŷ) against actual           │
  │       annotated labels (y) to extract the loss.             │
  │                                                             │
  │  Step 4: Backpropagation & Weight Updates                   │
  │     • Compute gradients and adjust weights using Gradient   │
  │       Descent (w = w − η · ∂L/∂w).                          │
  │                                                             │
  │  Step 5: Epoch Iterations                                   │
  │     • Repeat the loop over multiple full data passes.       │
  └─────────────────────────────────────────────────────────────┘
    

1. Weight & Bias Initialization: Connection weights (w) and biases (b) are initialized using random number generation.
2. Forward Pass: Training data inputs enter through the input layer and flow forward toward the output layer, where the network calculates its predictions.
3. Loss Computation: The network's predicted output values are compared against the ground-truth annotated labels provided in the training data to calculate the loss.
4. Backpropagation & Parameter Adjustments: The loss gradient is calculated, and the network uses the Gradient Descent algorithm to adjust weights and biases across all layers to reduce prediction error.
5. Iterative Epoch Loops: Steps 2 through 4 are repeated across multiple full dataset iterations called epochs until the loss function is minimized and accuracy is maximized.

3. Data Splitting Strategies for Generalization

To evaluate how well a model generalizes to new, unseen data, the raw dataset is split into distinct subsets.

  ┌─────────────────────────────────────────────────────────────┐
  │            Data Splitting Ratios & Set Roles                │
  ├───────────────────────────────┬─────────────────────────────┤
  │ 3-Set Split (Industry Rule)   │ Functional Role & Purpose   │
  ├───────────────────────────────┼─────────────────────────────┤
  │ • Training Set  : 60%         │ • Used to learn weights &   │
  │                               │   biases.                   │
  │ • Validation Set: 20%         │ • Used to tune hyper-       │
  │                               │   parameters.               │
  │ • Testing Set   : 20%         │ • Used to evaluate final    │
  │                               │   unseen performance.       │
  ├───────────────────────────────┼─────────────────────────────┤
  │ 2-Set Alternative Split       │                             │
  │ • Training Set  : 70%         │ • Learns parameters.        │
  │ • Testing Set   : 30%         │ • Tests final accuracy.     │
  └───────────────────────────────┴─────────────────────────────┘
    

A. Set Definitions & Roles:

• Training Set: Used directly by the network to learn optimal weights and biases during training.
• Validation Set: Used by data scientists to tweak and tune hyper-parameters.
• Testing Set: Kept completely separate during training; used to measure final model performance and testing error on unseen data.

B. Industry Rules of Thumb for Data Ratios:

• 3-Set Partitioning: 60% Training, 20% Validation, and 20% Testing.
• 2-Set Partitioning: 70% Training and 30% Testing.

4. Epochs, Batches, and Mini-Batch Training

When datasets contain large volumes of sample data, training is structured using epochs and batches:

  ┌─────────────────────────────────────────────────────────────┐
  │              Epoch vs. Batch vs. Mini-Batch                 │
  ├─────────────────────────────────────────────────────────────┤
  │  • Epoch: A single complete pass over the ENTIRE training   │
  │    dataset.                                                 │
  │                                                             │
  │  • Batch: Dividing a large dataset into smaller sample      │
  │    subsets.                                                 │
  │                                                             │
  │  • Mini-Batch Training: Training the network batch-by-batch │
  │    to improve training stability and computational speed.   │
  └─────────────────────────────────────────────────────────────┘
    

Concrete Numerical Example:

Suppose a training dataset contains 1,000 total samples:

• Selected Batch Size: 100 samples.
• Total Batches Generated: 1000 / 100 = 10 batches.
• Execution: Processing all 10 batches through a forward and backward pass once constitutes 1 complete Epoch.

5. Parameters vs. Hyper-parameters & Learning Rate (η) Dynamics

Understanding the distinction between model parameters and hyper-parameters is essential in neural network engineering:

  ┌─────────────────────────────────────────────────────────────┐
  │              Parameters vs. Hyper-parameters                │
  ├───────────────────────────────┬─────────────────────────────┤
  │ Category                      │ Definition & Examples       │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Model Parameters              │ Variables learned           │
  │                               │ AUTOMATICALLY by the network│
  │                               │ (e.g., Weights & Biases).   │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Hyper-parameters              │ Variables set MANUALLY by   │
  │                               │ humans before training      │
  │                               │ (e.g., Learning Rate η,     │
  │                               │ Batch Size, Epochs).        │
  └───────────────────────────────┴─────────────────────────────┘
    

The Weight Update Equation:

The learning rate (η) is a hyper-parameter that controls the step size of weight updates during gradient descent:

w = w − η · ∂L/∂w

Where η represents the learning rate, and ∂L/∂w represents the error gradient with respect to weight w.

  ┌─────────────────────────────────────────────────────────────┐
  │            Learning Rate (η) Impact Analysis                │
  ├───────────────────────────────┬─────────────────────────────┤
  │ Learning Rate Setting         │ Behavioral Impact & Results │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Too High (η = 0.1)            │ Takes 10% steps; overshoots │
  │                               │ target, causing unstable,   │
  │                               │ divergent training.         │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Too Low (η = 0.0001)          │ Learning is extremely slow; │
  │                               │ may fail to converge within │
  │                               │ allocated epochs.           │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Optimal (η = 0.01)            │ Fast learning with smooth,  │
  │                               │ stable convergence.         │
  └───────────────────────────────┴─────────────────────────────┘
    

Numerical Comparison of Learning Rates:

• High Learning Rate (η = 0.1): Makes 10% updates per step. This step size is too large, causing the model to overshoot target values, destabilize training, and diverge.
• Excessively Low Learning Rate (η = 0.0001): Makes tiny updates. Learning becomes sluggish and takes excessive compute time, failing to converge within specified epochs.
• Optimal Learning Rate (η = 0.01): Provides fast training while ensuring smooth, stable convergence. Recommended starting values are 0.01 or 0.001.

6. Model Evaluation Metrics & Post-Deployment Generalization

After training completes, the model must be evaluated to ensure it performs well on unseen client data post-deployment.

  ┌─────────────────────────────────────────────────────────────┐
  │              Evaluation Metrics by Task Type                │
  ├───────────────────────────────┬─────────────────────────────┤
  │ Task Type                     │ Evaluation Metrics Used     │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Classification Tasks          │ • Accuracy                  │
  │                               │ • Precision                 │
  │                               │ • Recall                    │
  │                               │ • F1-Score                  │
  │                               │ • Confusion Matrix          │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Regression Tasks              │ • Mean Squared Error (MSE)  │
  │                               │ • Mean Absolute Error (MAE) │
  │                               │ • Root Mean Squared Error   │
  │                               │   (RMSE)                    │
  └───────────────────────────────┴─────────────────────────────┘
    

• Confusion Matrix: Visualizes classification performance by showing how frequently the model confuses one category with another.
• Post-Deployment Performance: Thorough evaluation ensures that the model maintains high accuracy and low error when deployed to process unseen client data in real-world applications.

7. Summary Table: Key Pipeline Concepts

Training Pipeline Stage	Operational Description	Key Rule / Formula
Weight Initialization	Weights and biases initialized using random number generation.	Random initialization
Forward Pass	Computes predicted outputs from input to output layer.	Forward propagation
Loss & Backpropagation	Compares predictions to labels; updates parameters via Gradient Descent.	w = w − η · ∂L/∂w
Data Partitioning	Splits dataset to evaluate generalization (60/20/20 or 70/30).	Train (60%), Val (20%), Test (20%)
Mini-Batching	Divides 1,000 samples into 100-sample batches (10 batches = 1 Epoch).	Improves stability & speedup
Parameters vs. Hyper-parameters	Parameters are learned automatically; hyper-parameters are set manually.	Parameters: w, b; Hyper-params: η, batch size
Optimal Learning Rate	Controls update step size; start with small values like 0.01 or 0.001.	Fast & smooth convergence