Computational Thinking Classes in Padua

Chapter 148 — Neural Network Learning Rate and Training Parameters Explained

1. Introduction & Core Training Challenges

When training an artificial neural network, achieving optimal model performance requires balancing learning parameters and model complexity. If a network is improperly configured or trained, it typically suffers from one of two primary training failure modes: Underfitting (Under-training) or Overfitting (Over-training).

2. Underfitting (Under-Training)

A. Definition & Identification

Underfitting (Under-training) occurs when a neural network fails to accurately identify and capture the underlying patterns within the training dataset.

B. Primary Causes of Underfitting

1. Too Few Epochs: Stopping the training process prematurely before the model has had sufficient iterations to learn the dataset's patterns (for example, halting training at 500 epochs when 1,000 epochs were required).
2. Overly Simple Network Architecture: Keeping the neural network architecture too simple by using too few hidden layers or too few neurons per layer.
3. Excessively High Learning Rate: Setting the learning rate too high causes the model to make aggressive jumps during optimization, preventing it from converging onto the desired target accuracy.
4. Insufficient Dataset Features: A domain-level issue where the dataset provided lacks the necessary features required to distinguish and identify underlying data patterns.

3. Overfitting (Over-Training)

A. Definition & Identification

Overfitting (Over-training) occurs when a model learns the training data too excessively, memorizing noise and specific details rather than learning generalizable patterns.

• Key Diagnostic Indicator: The model exhibits very low training error but suffers from very high testing error on new, unseen data.

B. Primary Causes of Overfitting

1. Running Too Many Epochs: Training the network for excessive iterations (for example, running 1,000 epochs when optimal pattern learning was achieved at 500 epochs).
2. Overly Complex Network Architecture: Creating an excessively complex network containing unnecessary hidden layers or an excessive number of neurons.
3. Insufficient Training Data Volume or Variance: Training on a dataset that lacks sufficient quantity or structural variance.
4. Lack of Regularization: Failing to apply regularization techniques during training, which are necessary to constrain equation complexity and prevent the model from growing overly complex.

4. Evaluating Model Performance: The Confusion Matrix

To diagnose underfitting, overfitting, and overall model performance on classification tasks, data scientists utilize evaluation metrics built from a Confusion Matrix.

  ┌─────────────────────────────────────────────────────────────┐
  │                 Binary Confusion Matrix Structure           │
  ├───────────────────────────────┬─────────────────────────────┤
  │                               │ Actual Positive Class       │ Actual Negative Class
  ├───────────────────────────────┼─────────────────────────────┼─────────────────────────────┤
  │ Predicted Positive Class      │ True Positive (TP)          │ False Positive (FP)         │
  │                               │ (Correct Prediction)        │ (Type I Error)              │
  ├───────────────────────────────┼─────────────────────────────┼─────────────────────────────┤
  │ Predicted Negative Class      │ False Negative (FN)         │ True Negative (TN)          │
  │                               │ (Type II Error)             │ (Correct Prediction)        │
  └───────────────────────────────┴─────────────────────────────┴─────────────────────────────┘
    

A. The 4 Core Outcomes of a Binary Classifier:

• True Positive (TP): The model correctly predicts the positive class when it is actually present.
• True Negative (TN): The model correctly predicts the negative class when it is actually absent.
• False Positive (FP): The model incorrectly predicts the positive class when it is actually absent (Error Type 1).
• False Negative (FN): The model incorrectly predicts the negative class when it is actually present (Error Type 2).

5. Performance Metrics & Mathematical Formulas

Using the four values from the Confusion Matrix, four primary performance metrics are derived:

  ┌─────────────────────────────────────────────────────────────┐
  │              Key Evaluation Metric Formulas                 │
  ├─────────────────────────────────────────────────────────────┤
  │  1. Accuracy:                                               │
  │     Formula: Accuracy = (TP + TN) / Total Dataset Samples   │
  │                                                             │
  │  2. Precision:                                              │
  │     Formula: Precision = TP / (TP + FP)                     │
  │                                                             │
  │  3. Recall:                                                 │
  │     Formula: Recall = TP / (TP + FN)                        │
  │                                                             │
  │  4. F1-Score (Unbiased Metric):                             │
  │     Harmonic mean utilizing both Precision and Recall.      │
  └─────────────────────────────────────────────────────────────┘
    

1. Accuracy: Measures the overall proportion of correct predictions across the entire dataset: Accuracy = (TP + TN) / Total Number of Samples
2. Precision: Measures the proportion of correctly predicted positive instances out of all instances predicted as positive: Precision = TP / (TP + FP)
3. Recall: Measures the proportion of correctly predicted positive instances out of all actual positive instances present in the data: Recall = TP / (TP + FN)
4. F1-Score: Represents an unbiased performance metric calculated as the harmonic mean of both Precision and Recall.

6. Numerical Benchmark Example

Consider a binary classification dataset containing 100 actual positive examples and 100 actual negative examples (200 total samples):

  ┌─────────────────────────────────────────────────────────────┐
  │              Numerical Confusion Matrix Example             │
  ├─────────────────────────────────────────────────────────────┤
  │  Classification Predictions Output:                         │
  │    • True Positives  (TP) = 90  (Correct Positive)          │
  │    • True Negatives  (TN) = 85  (Correct Negative)          │
  │    • False Positives (FP) = 15  (Error)                     │
  │    • False Negatives (FN) = 10  (Error)                     │
  │                                                             │
  │  Calculated Metric Values:                                  │
  │    • Accuracy  = (90 + 85) / 200 = 175 / 200 = 87.5%        │
  │    • Precision = 90 / (90 + 15)   = 90 / 105  ≈ 85.7%       │
  │    • Recall    = 90 / (90 + 10)   = 90 / 100  = 90.0%       │
  └─────────────────────────────────────────────────────────────┘
    

• Correct Predictions: TP = 90 and TN = 85 (175 total correct predictions).
• Errors: FN = 10 and FP = 15 (25 total error predictions).
• Metric Calculations:
  • Accuracy: (90 + 85) / 200 = 175 / 200 = 87.5%
  • Precision: 90 / (90 + 15) = 90 / 105 ≈ 85.7%
  • Recall: 90 / (90 + 10) = 90 / 100 = 90.0%

7. Summary Comparison: Underfitting vs. Overfitting

Feature Dimension	Underfitting (Under-Training)	Overfitting (Over-Training)
Core Problem	Model fails to identify underlying data patterns.	Model memorizes training noise and specifics.
Epoch Count Cause	Too few training epochs.	Too many training epochs.
Architecture Cause	Network is too simple (too few layers/neurons).	Network is overly complex (too many layers/neurons).
Learning Rate Role	Learning rate is set too high (jumps/fails to converge).	Lack of regularization during optimization.
Diagnostic Behavior	High error on both training and testing sets.	Very low training error, but very high testing error.
Generalization	Fails to capture general dataset trends.	Fails to generalize to new, unseen examples.