Chapter 14: Recurrent Neural Networks (RNNs) & LSTMs

Bloom's Taxonomy Map for This Chapter

Learning Objectives

By the end of this chapter, you will be able to:

• Explain why sequential data (time series, text, speech) requires architectures with temporal memory — and why CNNs and feedforward networks fundamentally cannot handle variable-length temporal dependencies
• Derive the vanilla RNN forward pass equations: h_t = tanh(W_hh · h_{t-1} + W_xh · x_t + b_h) and explain weight sharing across time steps
• Perform Backpropagation Through Time (BPTT) — unroll the computation graph, apply the chain rule across T time steps, and compute ∂L/∂W_hh explicitly
• Prove the vanishing gradient problem: show that ∂h_T/∂h_1 = ∏_{t=1}^{T-1} diag(tanh'(z_t)) · W_hh decays exponentially when the largest singular value of W_hh < 1
• Derive each LSTM gate from first principles — forget gate (what to erase), input gate (what to write), candidate cell (what to write), output gate (what to read) — and show how the cell state provides a gradient highway
• Compare GRU's simplified two-gate architecture (update + reset) with LSTM's four gates, and identify when each is preferred
• Explain bidirectional RNNs (reading sequences forward + backward) and deep/stacked RNNs (multiple recurrent layers)
• Implement both RNN and LSTM cells from scratch in NumPy, including forward pass and BPTT
• Build practical models using PyTorch's nn.LSTM — character-level language model and time-series forecasting
• Preview the encoder-decoder (seq2seq) architecture that leads to attention and Transformers (Chapter 15)

Opening Hook — 25 Lakh Tickets and the Problem of Memory

The Intuition First — Thinking Before Equations

The Diary Analogy

Imagine you're reading a diary, one page per day. On each page, the writer describes what happened. When you read Day 5, you don't forget Days 1-4 — you carry a mental summary of everything you've read so far. This mental summary gets updated as you read each new page.

That's exactly what an RNN does:

• Each page = one input at time step t (today's features: bookings, weather, festival flag)
• Your mental summary = the hidden state h_t (a vector summarizing everything seen so far)
• Updating your summary = applying a function: new_summary = f(old_summary, today's page)
• Making a prediction = using today's summary to guess what tomorrow's page will say

The "Aha!" Question

If the RNN uses the same weights at every time step, how can it possibly distinguish between "the booking happened yesterday" vs. "the booking happened 30 days ago"?

Answer: It can't — not well, anyway. That's the vanishing gradient problem, and it's exactly why we need LSTMs. But before we jump to the solution, let's deeply understand the problem.

Mathematical Foundation — Deriving Everything from Scratch

PyTorch
# 3-layer LSTM with dropout
lstm = nn.LSTM(input_size=100, hidden_size=256,
               num_layers=3, dropout=0.3, batch_first=True)

Worked Examples

Example 1: RNN Forward Pass by Hand

Let's trace through a vanilla RNN with d_h = 2, d_x = 2, processing a 3-step sequence.

Example 2: IRCTC Demand Forecasting with LSTM (Indian Industry)

Example 3: Google Translate — Encoder-Decoder LSTM (US/Global)

Python Implementation — From Scratch (NumPy)

6.1 — Vanilla RNN Class

Python / NumPy
import numpy as np

class VanillaRNN:
    """Vanilla RNN implemented from scratch with BPTT."""

    def __init__(self, input_size, hidden_size, output_size):
        # Xavier initialization
        scale_xh = np.sqrt(2.0 / (input_size + hidden_size))
        scale_hh = np.sqrt(2.0 / (hidden_size + hidden_size))
        scale_hy = np.sqrt(2.0 / (hidden_size + output_size))

        self.W_xh = np.random.randn(hidden_size, input_size) * scale_xh
        self.W_hh = np.random.randn(hidden_size, hidden_size) * scale_hh
        self.W_hy = np.random.randn(output_size, hidden_size) * scale_hy
        self.b_h  = np.zeros((hidden_size, 1))
        self.b_y  = np.zeros((output_size, 1))

    def forward(self, inputs, h_prev):
        """
        inputs: list of T input vectors, each (input_size, 1)
        h_prev: initial hidden state (hidden_size, 1)
        Returns: outputs (list of T output vectors), hiddens, h_last
        """
        hiddens = {-1: h_prev.copy()}  # store for BPTT
        outputs = []

        for t, x_t in enumerate(inputs):
            # RNN forward: h_t = tanh(W_hh · h_{t-1} + W_xh · x_t + b_h)
            z_t = self.W_hh @ hiddens[t-1] + self.W_xh @ x_t + self.b_h
            h_t = np.tanh(z_t)
            hiddens[t] = h_t

            # Output: y_t = softmax(W_hy · h_t + b_y)
            logits = self.W_hy @ h_t + self.b_y
            probs = np.exp(logits) / np.sum(np.exp(logits))  # softmax
            outputs.append(probs)

        return outputs, hiddens, hiddens[len(inputs)-1]

    def bptt(self, inputs, targets, hiddens, outputs):
        """Backpropagation Through Time — compute all gradients."""
        T = len(inputs)
        # Initialize gradients
        dW_xh = np.zeros_like(self.W_xh)
        dW_hh = np.zeros_like(self.W_hh)
        dW_hy = np.zeros_like(self.W_hy)
        db_h  = np.zeros_like(self.b_h)
        db_y  = np.zeros_like(self.b_y)

        dh_next = np.zeros_like(self.b_h)  # gradient from future
        loss = 0.0

        for t in reversed(range(T)):
            # Cross-entropy loss gradient: dy = probs - one_hot(target)
            dy = outputs[t].copy()
            dy[targets[t]] -= 1.0
            loss += -np.log(outputs[t][targets[t], 0] + 1e-8)

            # Output layer gradients
            dW_hy += dy @ hiddens[t].T
            db_y  += dy

            # Gradient into hidden state
            dh = self.W_hy.T @ dy + dh_next

            # Backprop through tanh: dtanh = (1 - h²) ⊙ dh
            dz = (1.0 - hiddens[t] ** 2) * dh

            # Accumulate gradients for shared weights
            dW_xh += dz @ inputs[t].T
            dW_hh += dz @ hiddens[t-1].T
            db_h  += dz

            # Pass gradient to previous time step
            dh_next = self.W_hh.T @ dz

        # Gradient clipping to prevent explosions
        for grad in [dW_xh, dW_hh, dW_hy, db_h, db_y]:
            np.clip(grad, -5, 5, out=grad)

        return loss, dW_xh, dW_hh, dW_hy, db_h, db_y

    def update(self, grads, lr=0.01):
        """SGD parameter update."""
        dW_xh, dW_hh, dW_hy, db_h, db_y = grads
        self.W_xh -= lr * dW_xh
        self.W_hh -= lr * dW_hh
        self.W_hy -= lr * dW_hy
        self.b_h  -= lr * db_h
        self.b_y  -= lr * db_y

6.2 — LSTM Class from Scratch

Python / NumPy
class LSTMCell:
    """Single LSTM cell implemented from scratch."""

    def __init__(self, input_size, hidden_size):
        self.d_h = hidden_size
        self.d_x = input_size
        concat_size = hidden_size + input_size

        # Initialize all 4 gate weight matrices
        scale = np.sqrt(2.0 / concat_size)
        self.W_f = np.random.randn(hidden_size, concat_size) * scale
        self.W_i = np.random.randn(hidden_size, concat_size) * scale
        self.W_c = np.random.randn(hidden_size, concat_size) * scale
        self.W_o = np.random.randn(hidden_size, concat_size) * scale

        # Biases — NOTE: forget gate bias initialized to 1.0!
        self.b_f = np.ones((hidden_size, 1))   # ← Important trick!
        self.b_i = np.zeros((hidden_size, 1))
        self.b_c = np.zeros((hidden_size, 1))
        self.b_o = np.zeros((hidden_size, 1))

    def sigmoid(self, x):
        return 1.0 / (1.0 + np.exp(-np.clip(x, -500, 500)))

    def forward_step(self, x_t, h_prev, C_prev):
        """Single time step forward pass.
        x_t:    (input_size, 1)
        h_prev: (hidden_size, 1)
        C_prev: (hidden_size, 1)
        Returns: h_t, C_t, cache (for backprop)
        """
        # Concatenate h_{t-1} and x_t
        concat = np.vstack([h_prev, x_t])  # (d_h + d_x, 1)

        # Forget gate: f_t = σ(W_f · [h_{t-1}; x_t] + b_f)
        f_t = self.sigmoid(self.W_f @ concat + self.b_f)

        # Input gate: i_t = σ(W_i · [h_{t-1}; x_t] + b_i)
        i_t = self.sigmoid(self.W_i @ concat + self.b_i)

        # Candidate: C̃_t = tanh(W_c · [h_{t-1}; x_t] + b_c)
        C_tilde = np.tanh(self.W_c @ concat + self.b_c)

        # Cell state update: C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t
        C_t = f_t * C_prev + i_t * C_tilde

        # Output gate: o_t = σ(W_o · [h_{t-1}; x_t] + b_o)
        o_t = self.sigmoid(self.W_o @ concat + self.b_o)

        # Hidden state: h_t = o_t ⊙ tanh(C_t)
        h_t = o_t * np.tanh(C_t)

        # Cache for BPTT
        cache = (x_t, h_prev, C_prev, concat, f_t, i_t, C_tilde, C_t, o_t, h_t)
        return h_t, C_t, cache

    def forward(self, inputs, h0=None, C0=None):
        """Process entire sequence."""
        T = len(inputs)
        h = h0 if h0 is not None else np.zeros((self.d_h, 1))
        C = C0 if C0 is not None else np.zeros((self.d_h, 1))

        hiddens, cells, caches = [], [], []
        for t in range(T):
            h, C, cache = self.forward_step(inputs[t], h, C)
            hiddens.append(h)
            cells.append(C)
            caches.append(cache)

        return hiddens, cells, caches

    def backward_step(self, dh_next, dC_next, cache):
        """Single time step backward pass."""
        x_t, h_prev, C_prev, concat, f_t, i_t, C_tilde, C_t, o_t, h_t = cache

        # Gradient through h_t = o_t ⊙ tanh(C_t)
        tanh_C = np.tanh(C_t)
        do = dh_next * tanh_C
        dC = dh_next * o_t * (1 - tanh_C**2) + dC_next

        # Gradient through C_t = f_t ⊙ C_{t-1} + i_t ⊙ C̃_t
        df = dC * C_prev
        di = dC * C_tilde
        dC_tilde = dC * i_t
        dC_prev = dC * f_t

        # Gradient through activations
        df_raw = df * f_t * (1 - f_t)        # sigmoid derivative
        di_raw = di * i_t * (1 - i_t)
        dC_tilde_raw = dC_tilde * (1 - C_tilde**2)  # tanh derivative
        do_raw = do * o_t * (1 - o_t)

        # Gradients for weights and biases
        dW_f = df_raw @ concat.T
        dW_i = di_raw @ concat.T
        dW_c = dC_tilde_raw @ concat.T
        dW_o = do_raw @ concat.T
        db_f, db_i, db_c, db_o = df_raw, di_raw, dC_tilde_raw, do_raw

        # Gradient flowing to previous time step
        d_concat = (self.W_f.T @ df_raw + self.W_i.T @ di_raw +
                    self.W_c.T @ dC_tilde_raw + self.W_o.T @ do_raw)
        dh_prev = d_concat[:self.d_h]
        dx_t = d_concat[self.d_h:]

        return dh_prev, dC_prev, dW_f, dW_i, dW_c, dW_o, db_f, db_i, db_c, db_o


# ── Quick test ──
np.random.seed(42)
lstm = LSTMCell(input_size=4, hidden_size=3)
inputs = [np.random.randn(4, 1) for _ in range(5)]
hiddens, cells, caches = lstm.forward(inputs)
print(f"Final hidden state shape: {hiddens[-1].shape}")
print(f"Final hidden state:\n{hiddens[-1].flatten()}")
print(f"Final cell state:\n{cells[-1].flatten()}")

6.3 — Character-Level Language Model on Hindi Text

Python / NumPy
# Character-level RNN trained on Hindi text
import numpy as np

# Sample Hindi text (Devanagari)
text = "नमस्ते भारत। यह एक गहन शिक्षण पाठ्यपुस्तक है। " * 50

# Build character vocabulary
chars = sorted(set(text))
vocab_size = len(chars)
char_to_idx = {ch: i for i, ch in enumerate(chars)}
idx_to_char = {i: ch for i, ch in enumerate(chars)}
print(f"Vocabulary size: {vocab_size} unique Hindi characters")

# Initialize RNN
hidden_size = 64
rnn = VanillaRNN(vocab_size, hidden_size, vocab_size)

# Training loop
seq_length = 25
learning_rate = 0.01
smooth_loss = -np.log(1.0 / vocab_size) * seq_length

for epoch in range(1000):
    h_prev = np.zeros((hidden_size, 1))

    for start in range(0, len(text) - seq_length - 1, seq_length):
        # Prepare one-hot encoded inputs and targets
        inputs_seq = []
        targets = []
        for i in range(seq_length):
            x = np.zeros((vocab_size, 1))
            x[char_to_idx[text[start + i]]] = 1.0
            inputs_seq.append(x)
            targets.append(char_to_idx[text[start + i + 1]])

        # Forward pass
        outputs, hiddens, h_prev = rnn.forward(inputs_seq, h_prev)

        # Backward pass (BPTT)
        loss, *grads = rnn.bptt(inputs_seq, targets, hiddens, outputs)
        smooth_loss = 0.999 * smooth_loss + 0.001 * loss

        # Update weights
        rnn.update(grads, lr=learning_rate)

    if epoch % 100 == 0:
        print(f"Epoch {epoch:4d} | Loss: {smooth_loss:.4f}")

print("Training complete!")

def rnn_forward(x_sequence, W_hh, W_xh, b):
    h = np.zeros((64, 1))
    for x_t in x_sequence:
        h = np.tanh(W_hh @ h + W_xh @ x_t + b)
        h = np.zeros((64, 1))  # Reset for next step
    return h

Python Implementation — PyTorch Library Version

7.1 — LSTM for Time-Series Forecasting (IRCTC Demand)

PyTorch
import torch
import torch.nn as nn
import numpy as np

class IRCTCDemandLSTM(nn.Module):
    """LSTM model for IRCTC daily booking prediction."""

    def __init__(self, input_size=8, hidden_size=128, num_layers=2,
                 dropout=0.2):
        super().__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        # Stacked LSTM with dropout between layers
        self.lstm = nn.LSTM(
            input_size=input_size,
            hidden_size=hidden_size,
            num_layers=num_layers,
            dropout=dropout,
            batch_first=True   # Input shape: (batch, seq_len, features)
        )

        # Fully connected output: predict tomorrow's bookings
        self.fc = nn.Sequential(
            nn.Linear(hidden_size, 64),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(64, 1)   # Single output: booking count
        )

        # Initialize forget gate bias to 1.0 (critical trick!)
        for name, param in self.lstm.named_parameters():
            if 'bias' in name:
                n = param.size(0)
                # LSTM bias is [b_i, b_f, b_c, b_o] concatenated
                param.data[n//4:n//2].fill_(1.0)  # forget gate bias = 1

    def forward(self, x):
        """
        x: (batch_size, seq_len=30, features=8)
        Returns: (batch_size, 1) — predicted bookings
        """
        # Initialize hidden state and cell state
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size,
                         device=x.device)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size,
                         device=x.device)

        # LSTM forward pass
        lstm_out, (h_n, c_n) = self.lstm(x, (h0, c0))
        # lstm_out shape: (batch, 30, 128) — all hidden states
        # h_n shape: (2, batch, 128) — final hidden state per layer

        # Use the last time step's output for prediction
        last_hidden = lstm_out[:, -1, :]  # (batch, 128)
        prediction = self.fc(last_hidden)  # (batch, 1)

        return prediction


# ── Training ──
model = IRCTCDemandLSTM()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()

# Synthetic data simulating IRCTC bookings
np.random.seed(42)
T_total = 365  # 1 year of daily data
seq_len = 30   # Look back 30 days

# Generate features: bookings with trend + seasonality + festivals
days = np.arange(T_total)
bookings = (15000 + 3000 * np.sin(2 * np.pi * days / 7)    # weekly
            + 5000 * np.sin(2 * np.pi * days / 365)  # yearly
            + np.random.randn(T_total) * 1000)           # noise

# Normalize
mean_b, std_b = bookings.mean(), bookings.std()
bookings_norm = (bookings - mean_b) / std_b

# Create sequences
X, y = [], []
for i in range(len(bookings_norm) - seq_len):
    features = np.column_stack([
        bookings_norm[i:i+seq_len],          # bookings
        np.sin(2*np.pi*days[i:i+seq_len]/7),  # day-of-week
        np.cos(2*np.pi*days[i:i+seq_len]/7),
        np.sin(2*np.pi*days[i:i+seq_len]/365),# month-of-year
        np.cos(2*np.pi*days[i:i+seq_len]/365),
        np.random.rand(seq_len),              # monsoon proxy
        np.random.rand(seq_len),              # festival proxy
        np.random.rand(seq_len),              # price proxy
    ])
    X.append(features)
    y.append(bookings_norm[i+seq_len])

X = torch.FloatTensor(np.array(X))
y = torch.FloatTensor(np.array(y)).unsqueeze(1)

# Train
model.train()
for epoch in range(100):
    pred = model(X)
    loss = criterion(pred, y)
    optimizer.zero_grad()
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    optimizer.step()
    if epoch % 20 == 0:
        print(f"Epoch {epoch:3d} | MSE Loss: {loss.item():.6f}")

print(f"\nModel parameters: {sum(p.numel() for p in model.parameters()):,}")

7.2 — Comparing RNN, LSTM, GRU in PyTorch

PyTorch
# Quick comparison of RNN variants
import torch.nn as nn

configs = {
    "Vanilla RNN": nn.RNN(input_size=8, hidden_size=128, num_layers=2,
                          batch_first=True),
    "LSTM":        nn.LSTM(input_size=8, hidden_size=128, num_layers=2,
                           batch_first=True),
    "GRU":         nn.GRU(input_size=8, hidden_size=128, num_layers=2,
                          batch_first=True),
    "BiLSTM":      nn.LSTM(input_size=8, hidden_size=128, num_layers=2,
                           batch_first=True, bidirectional=True),
}

print(f"{'Model':<15} {'Parameters':>12} {'Output Shape':>18}")
print("─" * 48)

x = torch.randn(32, 30, 8)  # batch=32, seq=30, features=8

for name, layer in configs.items():
    params = sum(p.numel() for p in layer.parameters())
    out = layer(x)[0]
    print(f"{name:<15} {params:>12,} {str(out.shape):>18}")

Visual Diagrams

8.1 — RNN Architectures Taxonomy

8.2 — LSTM vs GRU Gate Comparison

8.3 — Gradient Flow: RNN vs LSTM

Common Misconceptions

GATE / Exam Corner

GATE Previous Year Questions (Pattern)

Interview Prep — India & US Focus

Conceptual Questions

Coding Questions

def lstm_forward(x_t, h_prev, c_prev, params):
    """Single LSTM step. All inputs are numpy arrays."""
    W_f, W_i, W_c, W_o = params['W_f'], params['W_i'], params['W_c'], params['W_o']
    b_f, b_i, b_c, b_o = params['b_f'], params['b_i'], params['b_c'], params['b_o']

    concat = np.concatenate([h_prev, x_t])       # [h; x]
    f = sigmoid(W_f @ concat + b_f)               # forget gate
    i = sigmoid(W_i @ concat + b_i)               # input gate
    c_tilde = np.tanh(W_c @ concat + b_c)         # candidate
    c = f * c_prev + i * c_tilde                   # cell update
    o = sigmoid(W_o @ concat + b_o)               # output gate
    h = o * np.tanh(c)                             # hidden state
    return h, c

Case Study Questions

Case Study — IRCTC Demand Forecasting (Indian Industry)

Case Study — Google Translate LSTM (US/Global Industry)

Hands-On Lab / Mini-Project

Project: Character-Level Hindi Text Generator with LSTM

Exercises — 22 Problems

Section A: Conceptual Questions (5)

Section B: Mathematical Problems (8)

Section C: Coding Problems (4)

def gru_cell(x_t, h_prev, params):
    Wr, Wz, Wh = params['Wr'], params['Wz'], params['Wh']
    br, bz, bh = params['br'], params['bz'], params['bh']
    concat = np.vstack([h_prev, x_t])
    r = sigmoid(Wr @ concat + br)         # reset gate
    z = sigmoid(Wz @ concat + bz)         # update gate
    concat_r = np.vstack([r * h_prev, x_t])
    h_tilde = np.tanh(Wh @ concat_r + bh)  # candidate
    h = z * h_prev + (1 - z) * h_tilde     # interpolate
    return h

# Clip by value
def clip_by_value(grad, c=5.0):
    return np.clip(grad, -c, c)

# Clip by norm (better method)
def clip_by_norm(grad, max_norm=5.0):
    norm = np.linalg.norm(grad)
    if norm > max_norm:
        grad = grad * (max_norm / norm)
    return grad

Section D: Critical Thinking (3)

★ Starred Research Problems (2)

Connections Map

Chapter Summary

Further Reading

🇮🇳 Indian Resources

• NPTEL: "Deep Learning" by Prof. Mitesh Khapra (IIT Madras) — Lectures 25-30 on RNNs and LSTMs, with excellent Hindi/English bilingual examples
• NPTEL: "Introduction to Machine Learning" by Prof. Sudeshna Sarkar (IIT Kharagpur) — Foundational sequence modeling
• GATE Previous Year Papers (CS/DA): ML section — RNN/LSTM questions appear regularly since GATE 2022
• CRIS Technical Reports: Railway demand forecasting methodologies used by Indian Railways

🌍 Global Resources

• Original Papers:
  • Hochreiter & Schmidhuber, "Long Short-Term Memory" (1997) — The LSTM paper
  • Cho et al., "Learning Phrase Representations using RNN Encoder-Decoder" (2014) — GRU introduction
  • Pascanu et al., "On the difficulty of training Recurrent Neural Networks" (2013) — Vanishing/exploding gradient analysis
  • Wu et al., "Google's Neural Machine Translation System" (2016) — GNMT architecture
• Interactive:
  • Distill.pub — "Understanding LSTM Networks" by Chris Olah (the gold standard visual explanation)
  • 3Blue1Brown — "Recurrent Neural Networks" (visual intuition)
  • Andrej Karpathy, "The Unreasonable Effectiveness of Recurrent Neural Networks" (2015 blog post)
• Textbooks:
  • Goodfellow, Bengio & Courville, "Deep Learning" — Chapter 10: Sequence Modeling
  • Jurafsky & Martin, "Speech and Language Processing" — RNN chapters
• Cutting Edge (2023-2025):
  • Gu & Dao, "Mamba: Linear-Time Sequence Modeling" (2023) — State-space models
  • Beck et al., "xLSTM: Extended Long Short-Term Memory" (2024) — LSTM revival
  • Peng et al., "RWKV: Reinventing RNNs for the Transformer Era" (2023)