Scratch Coding Classes for Kids in Miskolc

Chapter 143 — Artificial Neural Networks Explained for Beginners

1. Introduction & Evolution of Artificial Neural Networks (ANNs)

Artificial Neural Networks (ANNs) are a foundational supervised learning technique in artificial intelligence invented several decades ago. At its core, an ANN is a mathematical model inspired by the structure and working mechanisms of the human brain.

  ┌─────────────────────────────────────────────────────────────┐
  │                 Evolution of AI Paradigms                   │
  ├─────────────────────────────────────────────────────────────┤
  │  Early AI (70–80 Years Ago): Rule-Based Systems             │
  │    • Relied on hardcoded logic & manual rules               │
  │    • Failed to handle complex, unstructured tasks           │
  │                                                             │
  │  Biological Paradigm Shift: Replicating the Human Brain     │
  │    • Human brain = billions of interconnected neurons       │
  │    • Simulating neurons in computers yields human-like AI   │
  │                                                             │
  │  Modern Era: Foundation of Deep Learning                    │
  │    • ANNs learn patterns automatically from input data      │
  │    • Powers modern deep learning & modern AI applications   │
  └─────────────────────────────────────────────────────────────┘
    

• Early Rule-Based Systems: When artificial intelligence began 70 to 80 years ago, systems relied strictly on rule-based logic. When rule-based models failed to handle complex real-world tasks, scientists looked to replicate the human brain—the most intelligent structure known in the universe.
• Simulating Biological Intelligence: Because human intelligence originates in the brain through billions of physical neurons, researchers hypothesized that simulating artificial neurons inside computers would create human-like machine intelligence.
• The Foundation of Modern Deep Learning: Today, ANNs serve as the foundational bedrock for deep learning and the broader artificial intelligence boom. ANNs automatically discover and learn complex patterns directly from raw training data.
• Core Real-World Applications: ANNs are applied across a wide variety of domain tasks, including Classification and Regression, Image Recognition, and Natural Language Processing (NLP).

2. Biological vs. Artificial Neurons

To understand how ANNs function mathematically, it is helpful to compare a biological neuron to an artificial neuron:

  ┌─────────────────────────────────────────────────────────────┐
  │               Biological vs. Artificial Neuron              │
  ├───────────────────────────────┬─────────────────────────────┤
  │ Biological Neuron (Brain)     │ Artificial Neuron (ANN)     │
  ├───────────────────────────────┼─────────────────────────────┤
  │ • Dendrites (Receive inputs)  │ • Input Features (x_i)      │
  │ • Synaptic Strengths          │ • Connection Weights (w_i)  │
  │ • Cell Body Processing        │ • Weighted Sum + Bias (b)   │
  │ • Axon (Fires output signal)  │ • Activation Function (σ)   │
  └───────────────────────────────┴─────────────────────────────┘
    

A. Biological Neuron Mechanics

In the human brain, a biological neuron receives incoming electrical signals through its dendrites (inputs). The cell body processes these signals and, if a certain threshold is reached, fires an output signal down its axon to neighboring neurons.

B. Artificial Neuron Mechanics

An artificial neuron converts this biological mechanism into a precise mathematical process:

1. Inputs (x_i): Receives numerical inputs (x_1, x_2, x_3, x_4, …) from neighboring neurons or raw input data.
2. Weights (w_i): Multiplies each input by an assigned connection weight (x_1 w_1, x_2 w_2, x_3 w_3, …).
   • Weight Analogy: Imagine consulting a group of advisors before making a decision. You assign higher weight to trusted, intelligent advisors and lower weight to less reliable ones. Similarly, weights in a neural network dictate how strongly or weakly an input influences the output signal.
3. Bias (b): Adds a bias value b to the weighted sum. Biases adjust the baseline shift and help pull the mean of the sums closer to zero.
4. Activation Function (σ): Passes the total weighted sum through an activation function to introduce non-linearity into the network output.

Mathematical Neuron Equation

Combining these steps yields the formal mathematical equation for a single artificial neuron:

ŷ = σ ( Σ_i x_i w_i + b )

Expanded: ŷ = σ (x_1 w_1 + x_2 w_2 + x_3 w_3 + … + b)

Where:

• x: Input feature values
• w: Connection weights
• b: Neuron bias term
• σ: Activation function
• ŷ: Final output predicted by the neuron

Key Training Goal: ANNs learn the optimal values for all weights (w) and biases (b) automatically by training iteratively on input datasets.

3. Network Architecture & Layer Structure

In an Artificial Neural Network, individual artificial neurons are organized systematically into layers, where outputs from one layer become the inputs to the next.

  ┌─────────────────────────────────────────────────────────────┐
  │              Layer Architecture of an ANN                   │
  ├─────────────────────────────────────────────────────────────┤
  │                                                             │
  │   Input Layer         Hidden Layers          Output Layer   │
  │    (External)        (Internal Logic)         (External)    │
  │   ┌─────────┐      ┌─────┐     ┌─────┐      ┌─────────┐     │
  │   │ Neuron  │─────►│Neuron────►│Neuron├─────►│ Neuron  │     │
  │   ├─────────┤      ├─────┤     ├─────┤      ├─────────┤     │
  │   │ Neuron  │─────►│Neuron────►│Neuron├─────►│ Neuron  │     │
  │   └─────────┘      └─────┘     └─────┘      └─────────┘     │
  │   Receives raw    Hidden from outside world   Outputs final │
  │    data inputs      (2–3 in standard ANNs;    predictions   │
  │                     dozens in Deep Learning)                │
  └─────────────────────────────────────────────────────────────┘
    

1. Input Layer: The first layer in the network. It receives raw input feature data directly from the external world.
2. Hidden Layers: Intermediate processing layers situated between the input and output layers. They are called "hidden" because their internal calculations are hidden from the outside world. Standard ANNs typically contain 2 to 3 hidden layers, whereas advanced deep learning models contain dozens of hidden layers.
3. Output Layer: The final layer of the network. It outputs the network's final prediction back to the external world.

Example Architecture: A standard simple ANN architecture consists of 1 Input Layer, 2 Hidden Layers, and 1 Output Layer.

4. Activation Functions & The Necessity of Non-Linearity

Without activation functions, a neural network is inherently a simple linear model performing only linear additions and multiplications.

  ┌─────────────────────────────────────────────────────────────┐
  │            Why Non-Linear Activation Functions Matter       │
  ├─────────────────────────────────────────────────────────────┤
  │  Without Activation Functions:                              │
  │    • Network collapses into a basic linear equation         │
  │    • Fails on real-world complex patterns                   │
  │                                                             │
  │  With Non-Linear Activation Functions:                      │
  │    • Introduces non-linearity into model transformations    │
  │    • Solves complex non-linear problems:                    │
  │      1. Face detection & recognition in images              │
  │      2. Non-linear time-series forecasting                  │
  │      3. Speech understanding & NLP                          │
  └─────────────────────────────────────────────────────────────┘
    

By applying non-linear activation functions, neural networks can learn complex non-linear problems, such as:

• Face detection and recognition in image processing
• Non-linear time-series forecasting in data science
• Speech understanding and natural language processing

5. Comparative Analysis of Key Activation Functions

The three most widely used activation functions in ANNs are Sigmoid, Tanh, and ReLU:

  ┌─────────────────────────────────────────────────────────────┐
  │                 Activation Function Comparison              │
  ├─────────────────────────────────────────────────────────────┤
  │  1. Sigmoid Function:                                       │
  │     • Formula : σ(x) = 1 / (1 + e⁻ˣ)                        │
  │     • Range   : (0 to 1)                                    │
  │     • Best For: Binary Classification (Yes/No outputs)      │
  │     • Drawback: Vanishing Gradient Issue                    │
  │                                                             │
  │  2. Tanh Function (Hyperbolic Tangent):                     │
  │     • Formula : tanh(x) = (eˣ - e⁻ˣ) / (eˣ + e⁻ˣ)            │
  │     • Range   : (-1 to +1) [Zero-Centered]                  │
  │     • Best For: Hidden Layers (Faster convergence)          │
  │     • Drawback: Vanishing Gradient Issue                    │
  │                                                             │
  │  3. ReLU Function (Rectified Linear Unit):                  │
  │     • Formula : f(x) = max(0, x)                            │
  │     • Range   : (0 to +∞)                                   │
  │     • Best For: Deep Learning Models (Fast training)        │
  │     • Drawback: Dying ReLU Problem                          │
  └─────────────────────────────────────────────────────────────┘
    

A. The Sigmoid Activation Function

• Mathematical Formula: σ(x) = 1 / (1 + e^−x)
• Output Range: 0 to 1
• Primary Use Case: Binary Classification tasks where the output represents a probability between two classes (e.g., Yes or No).
• Graph Characteristics: Forms a smooth, S-shaped curve that transitions smoothly between 0 and 1 (unlike a harsh step function).
• Major Drawback: Suffers from the Vanishing Gradient Problem in deep networks, where learning gradients shrink close to zero and drastically slow down model training.

B. The Tanh (Hyperbolic Tangent) Function

• Mathematical Formula: tanh(x) = (e^x − e^−x) / (e^x + e^−x)
• Output Range: −1 to +1
• Primary Use Case: Excellent performance when applied to hidden layers.
• Graph Characteristics: Forms an S-shaped curve that is zero-centered (centered at 0), providing faster convergence during training compared to Sigmoid.
• Major Drawback: Still susceptible to the Vanishing Gradient Problem.

C. The ReLU (Rectified Linear Unit) Function

• Mathematical Formula: f(x) = max(0, x)
• Output Range: 0 to +∞ (Positive Infinity)
• Primary Use Case: The most popular activation function in deep learning.
• Graph Characteristics: Behaves linearly for positive values (x > 0) and truncates all negative values (x < 0) to zero.
• Key Advantages: Delivers extremely fast training and resolves the vanishing gradient problem.
• Major Drawback: Can suffer from the Dying ReLU Problem, where neurons get permanently stuck outputting zero during training.

6. Summary Comparison Table: Activation Functions

Activation Function	Mathematical Formula	Output Range	Ideal Placement / Best Use Case	Primary Limitation / Risk
Sigmoid	1 / (1 + e−x)	0 to 1	Output layer for Binary Classification	Vanishing Gradient Problem
Tanh	(ex − e−x) / (ex + e−x)	−1 to +1	Hidden Layers (Zero-centered convergence)	Vanishing Gradient Problem
ReLU	max(0, x)	0 to +∞	Deep Learning Models (Fast training)	Dying ReLU Problem