AI Courses for Kids in Klaipeda

Chapter 158 — LSTM Networks Explained: How AI Learns from Sequences and Time

1. Introduction: Why Do We Need LSTMs? (The RNN Memory Problem)

In traditional Recurrent Neural Networks (RNNs), processing sequential inputs works well over short sequences. However, standard RNNs suffer from a severe technical limitation known as the Vanishing Gradient Problem. As sequence lengths increase, error gradients propagated backward through time decay exponentially toward zero, causing weight updates to stall and preventing the network from learning long-term dependencies.

  ┌─────────────────────────────────────────────────────────────┐
  │                 Human Memory vs. LSTM Memory                │
  ├─────────────────────────────────────────────────────────────┤
  │  Human Memory System:                                       │
  │    • Short-Term Memory : Retains immediate recent details.  │
  │    • Long-Term Memory  : Selective retention—keeps crucial  │
  │                          memories and forgets irrelevant    │
  │                          noise over extended time.          │
  │                                                             │
  │  LSTM Neural Architecture:                                  │
  │    • Simulates human memory by combining both short-term    │
  │      and long-term memory streams into a single network.    │
  └─────────────────────────────────────────────────────────────┘
    

The Human Memory Analogy

Humans possess both short-term memory and long-term memory. Over extended periods, human long-term memory selectively retains crucial, meaningful information while systematically forgetting irrelevant or unimportant noise to prevent information overload.

To bridge this gap in artificial intelligence, Long Short-Term Memory (LSTM) networks were designed to simulate this dual-memory mechanism, enabling models to preserve key information across extended sequences.

2. Core Architectural Innovation: Memory Cells & The Conveyor Belt

An LSTM overcomes the limitations of standard RNNs by introducing a specialized Memory Cell regulated by a Gating Mechanism.

  ┌─────────────────────────────────────────────────────────────┐
  │                   LSTM Memory Architecture                  │
  ├─────────────────────────────────────────────────────────────┤
  │                                                             │
  │   Cell State (C_t)  ───► [ Conveyor Belt / Long-Term Memory ] ───►
  │                             ▲                    ▲          │
  │                             │ (Forget/Input)     │ (Output) │
  │                             │                    │          │
  │   Hidden State (h_t) ───► [ Short-Term Memory / Output ]  ───►
  │                                                             │
  └─────────────────────────────────────────────────────────────┘
    

A. Key Structural Components:

1. Cell State (C_t): Acts as the network's long-term memory buffer or conveyor belt. It runs straight through the sequence with minimal linear interaction, allowing information to flow mostly unchanged across long time spans unless modified by gates.
2. Hidden State (h_t): Represents the network's short-term memory and working output at time step t.
3. Gating Mechanism: Internal control structures that regulate the flow of information into and out of the cell state.

B. How Gates Control Information (The Sigmoid Filter):

Every gate in an LSTM employs a Sigmoid (σ) activation layer that outputs values between 0 and 1:

• 0 (Block): Completely blocks information from passing through.
• 1 (Allow): Allows information to flow through completely.
• Intermediate values (e.g., 0.65) allow partial information retention.

3. The 3 Gates & Candidate State (Step-by-Step Operations)

An LSTM cell executes four mathematical steps at every sequence time step t:

  ┌─────────────────────────────────────────────────────────────┐
  │                 4-Step LSTM Calculation Loop                │
  ├─────────────────────────────────────────────────────────────┤
  │  Step 1: Forget Gate (f_t)                                  │
  │     • Decides what old information to discard from C_{t-1}  │
  │                                                             │
  │  Step 2: Input Gate (i_t) & Candidate State (C̃_t)         │
  │     • Decides what new information to add to C_t            │
  │                                                             │
  │  Step 3: Update Cell State (C_t)                            │
  │     • Combines forgotten old memory with scaled new candidates│
  │                                                             │
  │  Step 4: Output Gate (o_t) & Hidden State (h_t)             │
  │     • Filters the cell state to emit short-term output h_t  │
  └─────────────────────────────────────────────────────────────┘
    

Step 1: The Forget Gate (f_t) — What to Throw Away

The Forget Gate determines which past information from the previous cell state (C_{t-1}) is no longer needed and should be discarded.

f_t = σ(W_f · [h_{t-1}, x_t] + b_f)

• Input: Previous hidden state h_{t-1} and current input x_t.
• Function: Evaluates features and outputs a scale factor between 0 (forget) and 1 (retain) for each value in the cell state.

Step 2: The Input Gate (i_t) & Candidate Cell State (C̃_t) — What to Store

The network next decides what new information to store in the cell state. This requires two sub-operations:

1. Input Gate (i_t): A Sigmoid layer that decides which values to update. i_t = σ(W_i · [h_{t-1}, x_t] + b_i)
2. Candidate Cell State (C̃_t): A tanh layer that creates a vector of new candidate values that could be added to the memory. C̃_t = tanh(W_c · [h_{t-1}, x_t] + b_c)

Step 3: Updating the Cell State (C_t) — Updating Memory

The network updates the old cell state C_{t-1} into the new cell state C_t by combining the results of the Forget Gate and Input Gate:

C_t = f_t · C_{t-1} + i_t · C̃_t

• f_t · C_{t-1}: Scales the old memory, selectively forgetting unwanted information.
• i_t · C̃_t: Scales the new candidate values, selectively adding relevant new information.

Step 4: The Output Gate (o_t) & Hidden State (h_t) — Generating Output

Finally, the network determines what output to emit in the current hidden state h_t based on the updated cell state:

1. Output Gate (o_t): A Sigmoid layer that decides which parts of the cell state to output. o_t = σ(W_o · [h_{t-1}, x_t] + b_o)
2. Hidden State Output (h_t): Passes the updated cell state C_t through tanh (to push values between -1 and 1) and multiplies it by the Output Gate: h_t = o_t · tanh(C_t)

4. Intuitive Example: Context Tracking in Natural Language

Consider processing a long news article:

"The engineer who built the cloud system grew up in France... [20 sentences describing career history]... She speaks fluent [French]."

  ┌─────────────────────────────────────────────────────────────┐
  │              Language Context Tracking Example              │
  ├─────────────────────────────────────────────────────────────┤
  │  1. Ingestion:                                              │
  │     • Input Gate detects "France" and stores country context│
  │       in Cell State (C_t).                                  │
  │                                                             │
  │  2. Long Sequence Traversal:                                │
  │     • Conveyor Belt passes "France" context unchanged across│
  │       20 intervening sentences describing career history.   │
  │                                                             │
  │  3. Topic Shift & Forget Gate:                              │
  │     • Forget Gate discards old subject context when topic   │
  │       switches.                                             │
  │                                                             │
  │  4. Output Prediction:                                      │
  │     • Output Gate extracts "France" context to predict      │
  │       target word "French".                                 │
  └─────────────────────────────────────────────────────────────┘
    

1. Input Gate Action: When reading "France", the Input Gate identifies it as an important location and stores this context in the Cell State (C_t).
2. Conveyor Belt Traversal: As the model processes 20 intervening sentences describing career history, the Cell State carries the "France" context unchanged through time.
3. Forget Gate Action: If the sentence subject changes (e.g., from singular to plural or country to country), the Forget Gate discards the outdated context.
4. Output Gate Action: When reaching "She speaks fluent...", the Output Gate queries the Cell State, retrieves "France", and accurately predicts "French".

5. Key Benefits & Real-World Applications

  ┌─────────────────────────────────────────────────────────────┐
  │                 Primary Application Domains                 │
  ├───────────────────────────────┬─────────────────────────────┤
  │ Application Domain            │ Operational Benefit         │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Natural Language Processing   │ Tracks document context     │
  │ (NLP)                         │ across long text passages.  │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Speech Recognition            │ Models continuous temporal  │
  │                               │ audio streams.              │
  ├───────────────────────────────┼─────────────────────────────┤
  │ Time-Series Forecasting       │ Captures long-term trends   │
  │                               │ in financial & weather data.│
  └───────────────────────────────┴─────────────────────────────┘
    

• Eliminates Vanishing Gradients: Constant linear flow along the cell state conveyor belt prevents gradients from vanishing, enabling learning across hundreds of time steps.
• Selective Information Control: Gates ensure the model stores essential context and discards irrelevant noise.
• Foundation for Advanced AI: Serves as a primary building block for sequence modeling across NLP, speech processing, and complex time-series analysis.

6. Summary Comparison: Standard RNN vs. LSTM Network

Feature / Aspect	Standard RNN Cell	Long Short-Term Memory (LSTM) Cell
Primary Limitation	Suffers severely from Vanishing Gradients.	Resolves Vanishing Gradients using Cell State.
Memory Capacity	Short-term memory only.	Dual Memory: Short-Term (h_t) & Long-Term (C_t).
Memory Flow	Single hidden state updated continuously.	Cell State Conveyor Belt preserves info unchanged.
Control Mechanism	No gating mechanism.	3 Gates: Forget (f_t), Input (i_t), Output (o_t).
Best Used For	Very short data sequences.	Long sequences (NLP, Speech, Time-Series).