Online Robotics Classes for Kids in Riga

Chapter 156 — Recurrent Neural Networks (RNNs) Explained for Beginners

1. Introduction: From Static Models to Sequential Data

In prior modules, we explored fundamental deep learning architectures such as Artificial Neural Networks (ANNs) and Convolutional Neural Networks (CNNs). While these networks excel at processing static data (such as tabular rows or standalone image files), real-world scenarios frequently involve sequential or time-dependent data.

  ┌─────────────────────────────────────────────────────────────┐
  │                 Static vs. Sequential Data                  │
  ├───────────────────────────────┬─────────────────────────────┤
  │ Static Data (ANNs / CNNs)     │ Sequential Data (RNNs)      │
  ├───────────────────────────────┼─────────────────────────────┤
  │ • Fixed-size independent      │ • Time-dependent order      │
  │   inputs (e.g., individual    │   where current data        │
  │   photos, single tabular      │   depends on previous data. │
  │   records).                   │                             │
  │ • Ignores temporal order      │ • Preserves sequence memory │
  │   and past history.           │   across time steps.        │
  └───────────────────────────────┴─────────────────────────────┘
    

What is Sequential Data?

Sequential data refers to information that arrives in a specific sequence or time order where each element depends on previous elements:

• Time-Series Data: Data points indexed with timestamps (e.g., daily stock prices, weather sensors).
• Text Data: Running streams of words in sentences, paragraphs, or articles.
• Speech Data: Continuous audio waveforms changing over time.
• Video Streams: Successive frame sequences forming continuous visual motion.

To process sequential data effectively, networks need an internal mechanism to remember past information—a capability provided by Recurrent Neural Networks (RNNs).

2. Core Architecture & Memory Loops

Unlike traditional feedforward networks that process inputs in a single forward sweep without internal loops, an RNN introduces recurrent loops that maintain an internal memory.

  ┌─────────────────────────────────────────────────────────────┐
  │                Recurrent Memory Loop Structure              │
  ├─────────────────────────────────────────────────────────────┤
  │                                                             │
  │     Current Input (x_t) ──┐                                 │
  │                           ▼                                 │
  │                    ┌─────────────┐                          │
  │  Previous Hidden ──► Hidden State│──► Current Output (y_t)  │
  │  State (h_{t-1})   │    (h_t)    │                          │
  │                    └──────┬──────┘                          │
  │                           │                                 │
  │                           └─► Passes Memory Loop to         │
  │                               Next Time Step (h_t)          │
  └─────────────────────────────────────────────────────────────┘
    

How Information Flows:

1. At each time step t, the network receives the current input vector (x_t) alongside the previous hidden state (h_{t-1}).
2. The hidden state acts as the network's memory, storing historical context from earlier time steps.
3. The output produced at time step t can serve as a prediction or be passed forward to influence calculations at the next time step t+1.

Flexible Input and Output Configurations:

RNNs can be configured to fit diverse sequential tasks:

• Sequence-to-Single Output: Predicts a single output from a full sequence (e.g., sentiment classification or time-series regression).
• Sequence-to-Sequence Output: Translates an input sequence into an output sequence (e.g., machine translation or text generation).

3. Mathematical Foundations & Layer Equations

At every time step t, the hidden state h_t and output y_t are updated using two primary equations:

  ┌─────────────────────────────────────────────────────────────┐
  │             Recurrent Neural Network Formulas               │
  ├─────────────────────────────────────────────────────────────┤
  │  1. Hidden State Update Equation:                           │
  │     h_t = σ( W_h · h_{t-1} + W_{xh} · x_t + b_h )           │
  │                                                             │
  │  2. Output Node Update Equation:                            │
  │     y_t = W_y · h_t + b_y                                   │
  └─────────────────────────────────────────────────────────────┘
    

Parameter Breakdown:

• x_t: Input vector at current time step t.
• h_{t-1}: Previous hidden state vector carrying past memory.
• h_t: Updated hidden state vector at current time step t.
• y_t: Generated output vector at current time step t.
• W_{xh}: Weight matrix applied to current input x_t.
• W_h: Weight matrix applied to previous hidden state h_{t-1}.
• W_y: Weight matrix applied to current hidden state h_t to generate output.
• b_h, b_y: Bias vectors for hidden and output layers.
• σ: Non-linear activation function (such as Tanh or ReLU).

4. Unrolling the RNN & Parameter Sharing

To visualize how an RNN operates across a sequence, we can "unroll" or expand the recurrent loop over successive time steps.

  ┌─────────────────────────────────────────────────────────────┐
  │                  Unrolled RNN Architecture                  │
  ├─────────────────────────────────────────────────────────────┤
  │                                                             │
  │    Output (y₁)           Output (y₂)           Output (y₃)  │
  │         ▲                     ▲                     ▲       │
  │         │ [W_y]               │ [W_y]               │ [W_y] │
  │    ┌────┴────┐  [W_h]    ┌────┴────┐  [W_h]    ┌────┴────┐  │
  │    │ Hidden  ├──────────►│ Hidden  ├──────────►│ Hidden  │  │
  │    │ State 1 │           │ State 2 │           │ State 3 │  │
  │    └────┬────┘           └────┬────┘           └────┬────┘  │
  │         ▲ [W_xh]              ▲ [W_xh]              ▲ [W_xh]│
  │         │                     │                     │       │
  │    Input (x₁)            Input (x₂)            Input (x₃)   │
  │                                                             │
  │   Key Insight: Identical Weights (W_h, W_xh, W_y) Reused    │
  │                Across ALL Time Steps (Parameter Sharing)    │
  └─────────────────────────────────────────────────────────────┘
    

The Principle of Parameter Sharing:

Unrolling the network reveals a critical efficiency property: the exact same weight matrices (W_{xh}, W_h, W_y) are reused across every time step. Rather than learning distinct weights for each step in a sequence, the network shares parameters across time, drastically reducing the total number of trainable parameters.

5. Real-World Applications & Examples

Recurrent Neural Networks are deployed across domains requiring sequence modeling:

  ┌─────────────────────────────────────────────────────────────┐
  │               Key RNN Real-World Applications               │
  ├───────────────────────────────┬─────────────────────────────┤
  │ Application Domain            │ Operational Mechanism       │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Stock Price Prediction        │ Predicts tomorrow's price   │
  │                               │ by processing past daily    │
  │                               │ price sequences.            │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Language Modeling & NLP       │ Predicts the next word in a │
  │                               │ sentence based on preceding │
  │                               │ context words.              │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Speech Recognition            │ Converts continuous audio   │
  │                               │ signals into text.          │
  └───────────────────────────────┴─────────────────────────────┘
    

• Stock Price Forecasting: Predicting tomorrow's stock price requires analyzing price sequences from previous days to identify trends.
• Language Modeling: Generating text or predicting missing words relies on surrounding context within running text sequences.
• Speech Recognition: Converting continuous voice recordings into written text requires tracking temporal audio patterns.

6. The Vanishing Gradient Problem & Limitations

When training RNNs using Backpropagation Through Time (BPTT), the network calculates gradients recursively across time steps. This process introduces a fundamental technical challenge known as the Vanishing Gradient Problem.

  ┌─────────────────────────────────────────────────────────────┐
  │           The Vanishing Gradient Problem Mechanism          │
  ├─────────────────────────────────────────────────────────────┤
  │  Cause:                                                     │
  │    • BPTT repeatedly multiplies partial derivatives during  │
  │      backward passes over many time steps.                  │
  │                                                             │
  │  Mathematical Effect:                                       │
  │    • Derivatives smaller than 1 shrink exponentially        │
  │      toward zero (e.g., 0.1 × 0.1 = 0.01).                  │
  │                                                             │
  │  Consequence:                                               │
  │    • Weight adjustments become infinitely small.            │
  │    • Learning stalls for long sequences.                    │
  │    • Network captures SHORT-TERM memory well but FAILS      │
  │      on LONG-TERM dependencies.                             │
  └─────────────────────────────────────────────────────────────┘
    

Why Gradients Vanish:

1. During BPTT, partial derivatives are multiplied repeatedly over time steps.
2. Multiplying numbers smaller than 1 causes values to shrink exponentially (e.g., 0.1 × 0.1 = 0.01).
3. Over long time sequences, these gradients approach zero, causing weight updates to freeze.
4. Consequently, basic RNNs excel at capturing short-term dependencies but perform poorly on long-term dependencies (e.g., long text documents or extended time series).

7. Summary Comparison Table

Feature / Metric	Feedforward Networks (ANNs / CNNs)	Recurrent Neural Networks (RNNs)
Data Compatibility	Static data (tables, standalone images).	Sequential / Time-series data (text, audio, video).
Data Connections	Strictly forward flow without feedback loops.	Contains recurrent loops that maintain memory.
Parameter Usage	Distinct weights per layer.	Parameter Sharing: Reuses identical weight matrices.
Hidden State Formula	h = Activation(W · x + b)	h_t = σ(W_h · h_{t-1} + W_{xh} · x_t + b_h)
Training Algorithm	Standard Backpropagation.	Backpropagation Through Time (BPTT).
Primary Limitation	Ignores temporal order in sequences.	Vanishing Gradient Problem on long sequences.