Python Coding Classes for Children in Milan

Chapter 151 — Neural Network Simulation: Understand How a Network Learns

1. Introduction & The Purpose of Interactive Neural Network Simulators

While the mathematical equations governing neural networks—such as weighted sums, activation functions, loss calculations, and backpropagation derivatives—are straightforward on paper, observing how these components interact in real-time can be challenging.

Interactive web-based neural network simulators solve this by providing a visual, hands-on environment. They allow students and AI engineers to inspect the inner mechanics of an Artificial Neural Network (ANN) step-by-step as it processes data, computes errors, updates connection weights, and shapes decision boundaries.

  ┌─────────────────────────────────────────────────────────────┐
  │             Two Primary Web Simulators Comparison           │
  ├───────────────────────────────┬─────────────────────────────┤
  │ Simulator 1: MLedict          │ Simulator 2: TensorFlow     │
  │ (mledict.com)                 │ Playground (tensorflow.org) │
  ├───────────────────────────────┼─────────────────────────────┤
  │ • Step-by-Step Arithmetic     │ • Real-Time Visualizer      │
  │ • Exposes exact formulas &    │ • Custom Architecture &     │
  │   numerical outputs per step  │   Hyperparameter Tuning     │
  │ • Ideal for tracing 10-step   │ • Ideal for observing       │
  │   forward & backward passes   │   decision boundary learning│
  └───────────────────────────────┴─────────────────────────────┘
    

1. Arithmetic Step-by-Step Visualizer (mledict.com): Lists all mathematical formulas, intermediate layer outputs, error derivatives, and weight adjustments across 10 distinct steps per training pass.
2. Advanced Interactive Visualizer (playground.tensorflow.org): A web platform for tuning hyperparameters (learning rate, activation function, regularization), adjusting hidden layers, and visualizing real-time decision boundary formation across diverse datasets.

2. The Non-Linear XOR Problem & Simulator Architecture

To demonstrate how a neural network learns, simulators often use the classic XOR (Exclusive OR) logic problem.

  ┌─────────────────────────────────────────────────────────────┐
  │                  The XOR Truth Table & Non-Linearity        │
  ├──────────────────────────┬──────────────────────────────────┤
  │ Input Features (x1, x2)  │ Target Output (y)                │
  ├──────────────────────────┼──────────────────────────────────┤
  │    x1 = 0, x2 = 0        │          y = 0                   │
  │    x1 = 1, x2 = 0        │          y = 1                   │
  │    x1 = 0, x2 = 1        │          y = 1                   │
  │    x1 = 1, x2 = 1        │          y = 0                   │
  └──────────────────────────┴──────────────────────────────────┘
    

Why XOR Requires a Neural Network:

• Non-Linear Separability: The XOR function outputs 1 if exactly one input is 1, and 0 if both inputs are identical (0 or 1). On a 2D graph, no single straight line can separate the 0 outputs from the 1 outputs.
• Failure of Linear Models: Simple linear classifiers (like single perceptrons without hidden layers) cannot solve XOR.
• Neural Network Solution: A Multi-Layer Perceptron (MLP) containing at least one hidden layer with non-linear activation functions easily learns the non-linear decision boundary required to solve XOR.

  ┌─────────────────────────────────────────────────────────────┐
  │                 3-Layer Simulator Architecture              │
  ├─────────────────────────────────────────────────────────────┤
  │                                                             │
  │   Input Layer             Hidden Layer          Output Layer│
  │  (2 Neurons)             (3 Neurons)             (1 Neuron) │
  │   ┌─────────┐            ┌─────────┐                        │
  │   │   x1    ├───────────►│ Neuron1 ├──┐                     │
  │   └─────────┘            ├─────────┤  │         ┌─────────┐ │
  │                          │ Neuron2 ├──┼────────►│  Output │ │
  │   ┌─────────┐            ├─────────┤  │         │   (y)   │ │
  │   │   x2    ├───────────►│ Neuron3 ├──┘         └─────────┘ │
  │   └─────────┘            └─────────┘                        │
  │                                                             │
  │   Initial State: All weights randomly initialized          │
  └─────────────────────────────────────────────────────────────┘
    

• Input Layer: 2 Neurons corresponding to input features x1 and x2.
• Hidden Layer: 3 Neurons equipped with non-linear activation functions.
• Output Layer: 1 Neuron emitting predicted class probability ŷ.
• Weights: Randomly initialized prior to training.

3. Detailed 10-Step Forward Pass and Backpropagation Execution

In an arithmetic simulator (e.g., mledict.com), executing a single training step breaks down into 10 sequential calculation steps across forward propagation, loss evaluation, and backpropagation:

  ┌─────────────────────────────────────────────────────────────┐
  │            10-Step Neural Network Execution Cycle           │
  ├─────────────────────────────────────────────────────────────┤
  │  FORWARD PASS:                                              │
  │  • Step 1 : Feed sample input row (e.g., x1 = 1, x2 = 1)   │
  │  • Step 2 : Compute Hidden Layer outputs (e.g., 0.35, 0.74, 0.53)│
  │  • Step 3 : Compute Output Layer prediction (e.g., y_hat = 0.25)│
  │  • Step 4 : Compute Cost / Loss (Target y = 0 vs. y_hat = 0.25)│
  │                                                             │
  │  BACKPROPAGATION PASS:                                      │
  │  • Step 5 : Compute Output Layer Error (e.g., Error = 0.049)│
  │  • Step 6 : Compute Hidden Layer Errors using derivatives   │
  │  • Step 7 : Compute Error w.r.t. Hidden-to-Output Weights   │
  │  • Step 8 : Compute Error w.r.t. Input-to-Hidden Weights    │
  │  • Step 9 : Update Hidden-to-Output Weights (Highlighted Red)│
  │  • Step 10: Update Input-to-Hidden Weights (Highlighted Green)│
  └─────────────────────────────────────────────────────────────┘
    

Step-by-Step Breakdown:

1. Step 1 (Input Ingestion): The simulator selects the first row from the dataset (x1 = 1, x2 = 1, target y = 0) and feeds x1 = 1 and x2 = 1 into the input layer.
2. Step 2 (Hidden Layer Forward Pass): The weighted sums plus biases are calculated for all 3 hidden neurons and passed through the activation function, outputting values such as 0.35, 0.74, and 0.53.
3. Step 3 (Output Layer Forward Pass): The hidden outputs are multiplied by hidden-to-output weights to produce the network's predicted output (ŷ = 0.25).
4. Step 4 (Cost Calculation): The loss function evaluates the error between the desired target (y = 0) and the predicted value (ŷ = 0.25).
5. Step 5 (Output Layer Backpropagation): Backpropagation begins by computing the error derivative at the output layer (e.g., Output Error = 0.049).
6. Step 6 (Hidden Layer Backpropagation): Using derivative chain rules, the output error is propagated backward to calculate error values across all 3 hidden layer neurons.
7. Step 7 (Hidden-to-Output Weight Gradients): Error gradients are calculated with respect to the weights connecting the hidden layer to the output layer.
8. Step 8 (Input-to-Hidden Weight Gradients): Error gradients are calculated with respect to the weights connecting the input layer to the hidden layer.
9. Step 9 (Update Hidden-to-Output Weights): Gradient descent updates the hidden-to-output weights (visualized as turning red in the simulator interface).
10. Step 10 (Update Input-to-Hidden Weights): Gradient descent updates the input-to-hidden weights (visualized as turning green in the simulator interface), completing one full forward and backward cycle.

4. Epochs, Iterations, and Convergence Analysis

  ┌─────────────────────────────────────────────────────────────┐
  │                 Steps vs. Iterations vs. Epochs             │
  ├─────────────────────────────────────────────────────────────┤
  │  • One Step Forward      : Advances 1 sub-step out of 10   │
  │                            for a single data row.          │
  │                                                             │
  │  • One Iteration (Epoch) : Executes all 10 steps across ALL │
  │                            4 XOR data rows.                │
  │                                                             │
  │  • 1000 Iterations      : Rapidly loops through 1,000      │
  │                            epochs to achieve convergence.  │
  └─────────────────────────────────────────────────────────────┘
    

• One Step Forward: Advances the simulation by a single computational step (e.g., moving from hidden layer calculation to output layer prediction) for one sample row.
• One Iteration Forward (Epoch): Executes the complete 10-step training pass across all 4 rows of the XOR dataset.
• 1000 Iterations Forward: Automatically runs 1,000 complete epochs in rapid succession.
• Convergence Result: After 1,000 iterations, the weights adjust from their initial random values to optimal values. Re-testing the input (x1 = 1, x2 = 1) yields a prediction of 0.49 (moving significantly closer toward the target 0 compared to initial untrained random outputs).

5. Advanced Controls in TensorFlow Playground

For higher-level architectural exploration, TensorFlow Playground (playground.tensorflow.org) provides extensive interactive controls:

  ┌─────────────────────────────────────────────────────────────┐
  │            TensorFlow Playground Configuration Panel         │
  ├───────────────────────────────┬─────────────────────────────┤
  │ Control Category              │ Available Tuning Parameters │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Hyperparameters               │ • Learning Rate             │
  │                               │ • Activation Function       │
  │                               │   (ReLU, Sigmoid, Tanh)     │
  │                               │ • Regularization (L1, L2)   │
  │                               │ • Regularization Rate       │
  │                               │ • Problem Type              │
  │                               │   (Classification/Regression)│
  ├───────────────────────────────┼─────────────────────────────┤
  │ Dataset Parameters            │ • Datasets (Circle, XOR,    │
  │                               │   Spiral, Gaussian)         │
  │                               │ • Train / Test Split Ratio  │
  │                               │ • Noise Level               │
  │                               │ • Batch Size                │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Architecture Customization    │ • Add/Remove Hidden Layers  │
  │                               │ • Add/Remove Neurons/Layer  │
  │                               │ • Feature Engineering (x1², │
  │                               │   x2², x1x2, sin(x1), etc.) │
  └───────────────────────────────┴─────────────────────────────┘
    

Key Features & Exploration Capabilities:

1. Hyperparameter Tuning: Test how changing the Learning Rate (e.g., 0.001 vs. 0.1) or switching Activation Functions (ReLU vs. Sigmoid vs. Tanh) accelerates or slows down convergence.
2. Regularization (L1 & L2): Apply regularization penalties to prevent overfitting and observe weight shrinkage in real time.
3. Dataset Complexity: Switch between datasets of varying difficulty—from simple linearly separable Gaussian clusters to complex non-linear patterns like Circles, XOR, and Intertwined Spirals.
4. Interactive Architecture Modification: Add or remove hidden layers and individual neurons dynamically using simple + and − buttons to discover the minimal network size required to solve a given dataset.
5. Real-Time Decision Boundary Visualization: As the simulation runs, the background colors update continuously to display the network's learned non-linear decision boundaries.

6. Summary Comparison Table

Simulator Feature	Step-by-Step Arithmetic Simulator (mledict.com)	Advanced Visual Simulator (playground.tensorflow.org)
Primary Focus	Mathematical calculations & exact formula outputs.	Hyperparameter tuning & visual decision boundaries.
Granularity	Step-by-step inspection of 10 forward & backward sub-steps.	Real-time continuous execution across full epochs.
Dataset Focus	Standardized XOR truth table (4 rows).	Circle, XOR, Gaussian, and Spiral datasets.
Weight Inspection	Displays numerical weight values & color-coded updates.	Displays visual connection thickness & color intensity.
Best Used For	Hand-calculation verification & deep internal understanding.	Intuitive architectural design & hyperparameter experimentation.