AI Courses for Kids in Turin

Chapter 145 — The Logistic (Sigmoid) Activation Function Explained

1. Introduction & The Purpose of Activation Functions

In Artificial Neural Networks (ANNs), activation functions play a vital role in determining whether an individual neuron should activate (emit an output signal) or remain inactive.

When input features pass through a neuron, the network first computes a weighted sum plus bias. The activation function is the final operation performed inside the neuron, taking this weighted sum as its input and transforming it into an output value.

  ┌─────────────────────────────────────────────────────────────┐
  │             Role of Activation Functions in ANNs            │
  ├─────────────────────────────────────────────────────────────┤
  │  1. Neuron Decision Operator:                               │
  │     • Decides whether a neuron fires/activates.            │
  │                                                             │
  │  2. Introduces Non-Linearity:                               │
  │     • Allows networks to learn complex non-linear patterns  │
  │       and non-linear decision boundaries.                   │
  │                                                             │
  │  3. Prevents Linear Collapse:                               │
  │     • Without activation functions, a multi-layer network   │
  │       collapses into a basic linear model restricted to     │
  │       straight-line separations.                            │
  └─────────────────────────────────────────────────────────────┘
    

Why Non-Linearity Matters:

• Linear Models: If no activation function is applied, a neural network simplifies into a standard linear model. Linear models can only solve linearly separable problems where categories can be divided by a straight line.
• Non-Linear Decision Boundaries: Applying non-linear activation functions introduces non-linear transformations. This enables the network to learn intricate curves and complex decision boundaries, allowing it to solve highly sophisticated real-world tasks.

2. Mathematical Definition & Formula

The Logistic Function—commonly referred to as the Sigmoid Activation Function—is historically one of the most important activation functions in machine learning.

It takes any real-valued input number across (−∞, +∞) and squashes it into a bounded continuous output range between 0 and 1.

  ┌─────────────────────────────────────────────────────────────┐
  │            The Sigmoid (Logistic) Formula & Curve           │
  ├─────────────────────────────────────────────────────────────┤
  │                                                             │
  │                   σ(x) = 1 / (1 + e⁻ˣ)                      │
  │                                                             │
  │  Where:                                                     │
  │    • x  = Weighted sum input value                          │
  │    • e  = Euler's number (base of natural logarithm ≈ 2.718)│
  │    • σ  = Output bounded strictly in the range (0, 1)       │
  └─────────────────────────────────────────────────────────────┘
    

Numerical Input-Output Examples:

Input Value (x)	Sigmoid Output σ(x)	Interpretation / Behavior
x = −3.0	0.047	Large negative input → Output close to 0
x = −1.0	0.269	Small negative input → Output < 0.5
x = 0.0	0.500	Zero input → Midpoint output (0.5)
x = +3.0	0.953	Large positive input → Output close to 1

• Large Positive Inputs (x → +∞): As x increases, e^−x approaches 0, driving the output σ(x) very close to 1.0.
• Large Negative Inputs (x → −∞): As x becomes deeply negative, e^−x grows rapidly toward +∞, driving the output σ(x) very close to 0.0.
• Zero Input (x = 0): At x = 0, e⁰ = 1, yielding an exact output of 0.5.

3. How the Sigmoid Function Operates Inside a Neuron

Inside an artificial neuron, the Sigmoid function acts as the final decision operator applied after the linear aggregation step:

  Inputs (x_i)        Weights (w_i) & Bias (b)         Weighted Sum (g)        Activation σ(g)
  ┌──────────┐        ┌──────────────────────┐        ┌────────────────┐       ┌──────────────┐
  │  x_1,    │  ───►  │  Multiply Weights &  │  ───►  │  g = w1*x1 +   │ ───►  │  a = σ(g) =  │ ──► Final
  │  x_2, ...│        │  Add Bias Term (b)   │        │      w2*x2 + b │       │  1/(1+e⁻ᵍ)   │     Prediction
  └──────────┘        └──────────────────────┘        └────────────────┘       └──────────────┘
    

1. Step 1: Weighted Sum Calculation (g): The neuron receives input features (x₁, x₂, …), multiplies them by their corresponding connection weights (w₁, w₂, …), and adds a bias term b: g = w₁x₁ + w₂x₂ + … + b
2. Step 2: Activation Pass (a): The resulting weighted sum g is passed directly into the Sigmoid function: a = σ(g) = 1 / (1 + e^−g)
3. Step 3: Final Output: The calculated scalar value a represents the neuron's final prediction or activation output.

4. Key Advantages of the Logistic (Sigmoid) Function

  ┌─────────────────────────────────────────────────────────────┐
  │                 Major Advantages of Sigmoid                 │
  ├─────────────────────────────────────────────────────────────┤
  │  1. Direct Probability Interpretation:                      │
  │     • Bounded output (0 to 1) matches probability range.    │
  │                                                             │
  │  2. Ideal for Binary Classification & Thresholding:         │
  │     • Easily thresholded (e.g., ≥ 0.5 → 1, < 0.5 → 0).      │
  │                                                             │
  │  3. Differentiable for Backpropagation:                     │
  │     • Smooth derivative enables gradient descent learning   │
  │       for updating weights and biases.                      │
  │                                                             │
  │  4. High Information Preservation in Small Networks:        │
  │     • Retains rich information in compact architectures.    │
  └─────────────────────────────────────────────────────────────┘
    

1. Natural Probability Mapping: Because probabilities natively range from 0 to 1, the output of a Sigmoid activation function can be directly interpreted as a predicted probability score.
2. Binary Classification & Decision Thresholding: The bounded output range (0 to 1) makes Sigmoid ideal for binary outcome modeling. Applying a decision threshold (such as 0.5) converts continuous outputs into discrete binary decisions: values above 0.5 are assigned to class 1, and values below 0.5 are assigned to class 0.
3. Differentiable for Backpropagation: The Sigmoid function is smooth and continuously differentiable. Differentiability is mandatory for the Backpropagation algorithm, which calculates gradients during gradient descent to train and update network weights and biases.
4. Information Preservation in Compact Networks: Historically, when compute power was limited and GPU acceleration was unavailable, neural networks had to remain small. The Sigmoid function preserves a high density of information, allowing smaller networks to solve complex problems effectively.

5. Limitations & Drawbacks

Despite its historical importance, the Sigmoid function exhibits three major technical limitations that restrict its usage in modern deep learning:

  ┌─────────────────────────────────────────────────────────────┐
  │               Key Limitations of Sigmoid                    │
  ├─────────────────────────────────────────────────────────────┤
  │  1. Vanishing Gradient Problem:                             │
  │     • Extremely large or small inputs cause gradients to    │
  │       shrink close to zero, stalling network learning.      │
  │                                                             │
  │  2. Not Zero-Centered (0.5-Centered):                       │
  │     • Outputs are centered at 0.5 rather than 0, making     │
  │       gradient optimization updates less efficient.         │
  │                                                             │
  │  3. Slow Convergence in Deep Architectures:                 │
  │     • Sluggish gradient updates lead to slow training       │
  │       times in networks with many hidden layers.            │
  └─────────────────────────────────────────────────────────────┘
    

• The Vanishing Gradient Problem: For very large positive or very large negative input values, the slope (derivative) of the Sigmoid curve flattens out and becomes near-zero. During backpropagation, multiplying these near-zero gradients across many layers causes the error gradient to "vanish", effectively halting weight updates and freezing network learning.
• Not Zero-Centered: The output of the Sigmoid function is centered at 0.5 rather than 0. Non-zero-centered outputs can cause systematic zig-zagging dynamics during gradient updates, slowing down optimization efficiency.
• Slow Convergence in Deep Networks: Because deep networks contain many layers and neurons requiring fast parameter updates, Sigmoid's small gradients lead to sluggish convergence.

6. Practical Design Guidelines: Where & When to Use Sigmoid

To optimize neural network performance, modern machine learning practitioners follow specific deployment rules regarding where to place the Sigmoid function:

  ┌─────────────────────────────────────────────────────────────┐
  │             Recommended Architectural Placement             │
  ├─────────────────────────────────────────────────────────────┤
  │  Output Layer (Binary Classification / Probability):        │
  │    • USE SIGMOID: Perfect for predicting probability scores │
  │      and binary outcomes (0 or 1).                          │
  │                                                             │
  │  Hidden Layers (Modern Deep Learning):                      │
  │    • AVOID SIGMOID: Replaced by ReLU (Rectified Linear Unit)│
  │      to prevent vanishing gradients and speed up training   │
  │      convergence.                                           │
  └─────────────────────────────────────────────────────────────┘
    

1. Output Layer for Binary Classification: Sigmoid is ideal when placed on the output layer of a network designed for binary classification or probability estimation.
2. Logistic Regression Models: Serves as the primary mathematical core for standalone Logistic Regression modeling.
3. Hidden Layers in Deep Networks: In modern deep learning, Sigmoid is rarely used in hidden layers. It has been largely replaced by functions like ReLU (Rectified Linear Unit), which eliminate vanishing gradients and provide drastically faster training convergence.

7. Summary Table: Logistic (Sigmoid) Function Overview

Aspect / Dimension	Description & Properties
Mathematical Formula	σ(x) = 1 / (1 + e−x)
Output Bounded Range	Continuous range from 0.0 to 1.0 (Centered at 0.5)
Key Output Values	x = −3 → 0.047; x = 0 → 0.5; x = +3 → 0.953
Primary Advantages	Output maps directly to probabilities; smooth & differentiable for backpropagation
Primary Limitations	Vanishing gradient problem; not zero-centered; slow convergence in deep networks
Best Architectural Role	Output Layer for binary classification and probability estimation
Hidden Layer Replacement	Replaced by ReLU in modern deep learning hidden layers