Chapter 14: LSTMs and GRUs — Solving Long-Term Memory

Bloom's Taxonomy Map for This Chapter

Learning Objectives

By the end of this chapter, you will be able to:

• Explain why vanilla RNNs fail on long sequences by deriving the vanishing gradient problem through repeated Jacobian multiplication
• Derive the complete LSTM cell equations — forget gate (f), input/update gate (i), cell candidate (c̃), cell state (c), output gate (o), and hidden state (a) — with full mathematical notation
• Derive the GRU equations — update gate (z), reset gate (r), candidate hidden state (h̃), and final hidden state (h) — and explain how GRU merges the forget and input gates
• Compare LSTM vs GRU on parameter count, training speed, and performance across different sequence lengths
• Explain Bidirectional RNNs — why reading a sequence both forwards and backwards helps tasks like Named Entity Recognition
• Implement an LSTM cell forward pass from scratch using only NumPy
• Build a TensorFlow LSTM model for NSE Nifty 50 stock price prediction using real-world financial time-series data
• Build a Bidirectional LSTM for Named Entity Recognition on Indian news articles
• Analyze the HDFC Bank case study — how LSTMs on transaction sequences reduced false positives in fraud detection by 40%
• Design deep/stacked RNN architectures and know when to add depth vs. width

Opening Hook — The Sentence That Broke the RNN

This chapter is about the most important architectural innovation in sequence modeling history: the gated recurrent cell. We'll study two variants — the LSTM (Long Short-Term Memory, 1997) and the GRU (Gated Recurrent Unit, 2014) — that solved the vanishing gradient problem and powered everything from Google Translate to Alexa to financial fraud detection at HDFC Bank.

Core Concepts

We begin by understanding why vanilla RNNs fail, then build the LSTM and GRU architectures gate by gate.

14.1 The Vanishing Gradient Problem in RNNs — Why Memory Fades

Recall from Chapter 13 that a vanilla RNN computes:

During backpropagation through time (BPTT), the gradient of the loss at time step T with respect to the hidden state at time step t requires computing:

This is a product of (T − t) matrices. Let's analyze what happens:

The key insight that leads to LSTM: we need a path where gradients can flow unchanged across many time steps. Instead of multiplying through W_aa repeatedly, we need an additive connection — a highway for gradients. This is the cell state.

14.2 LSTM — Long Short-Term Memory (Full Derivation)

The LSTM, introduced by Hochreiter & Schmidhuber (1997) and refined by Gers et al. (2000) with the forget gate, replaces the simple RNN hidden state with a carefully engineered memory cell controlled by three gates.

The Key Idea: Two Separate State Vectors

Unlike a vanilla RNN (which has only one hidden state a⟨t⟩), an LSTM maintains two vectors at each time step:

• Cell state c⟨t⟩ — the "long-term memory" that flows along a conveyor belt with minimal modification
• Hidden state a⟨t⟩ (sometimes written h⟨t⟩) — the "working memory" exposed to the outside world

Step-by-Step Gate Derivation

At each time step t, the LSTM receives three inputs: the previous hidden state a⟨t−1⟩, the previous cell state c⟨t−1⟩, and the current input x⟨t⟩. It produces updated c⟨t⟩ and a⟨t⟩.

Gate 1: Forget Gate (f⟨t⟩) — "What to erase from memory"

The forget gate outputs a vector of values between 0 and 1 (sigmoid output). Each element decides how much of the corresponding cell state dimension to retain:

• f = 1: Keep this memory completely (e.g., "remember that this is a question")
• f = 0: Erase this memory completely (e.g., "forget the previous subject, new subject introduced")
• f = 0.7: Keep 70% of this memory — gradual decay

Gate 2: Input/Update Gate (i⟨t⟩) — "What new information to store"

The input gate decides which dimensions of the cell state will receive new information. Like the forget gate, it outputs values in [0, 1].

Cell Candidate (c̃⟨t⟩) — "What new information to potentially store"

The cell candidate is the proposed new memory content. It uses tanh (output in [−1, 1]) because cell state values can be positive or negative. Think of this as the "raw new information" — the input gate decides how much of it to actually write.

Cell State Update (c⟨t⟩) — "The actual memory update"

This is the most important equation in the LSTM. The ⊙ symbol denotes element-wise (Hadamard) multiplication. Notice:

• The first term f⟨t⟩ ⊙ c⟨t−1⟩ selectively forgets parts of the old memory
• The second term i⟨t⟩ ⊙ c̃⟨t⟩ selectively writes new information
• The cell state update is additive (not multiplicative!) — this is why gradients flow easily through time

Gate 3: Output Gate (o⟨t⟩) — "What to reveal from memory"

Hidden State (a⟨t⟩) — "The visible output"

The hidden state is a filtered version of the cell state. The cell state might store "this is a question sentence" and "the subject is aap", but at the current time step, only the relevant information is revealed through the output gate.

Complete LSTM Equations — Summary

LSTM Parameter Count

Let n = hidden size and m = input size. Each gate has a weight matrix of shape (n, n+m) and a bias of shape (n,). With 4 sets of parameters (forget, input, cell candidate, output):

For n = 256, m = 100: Total = 4(256²) + 4(256)(100) + 4(256) = 262,144 + 102,400 + 1,024 = 365,568 parameters per LSTM layer.

14.3 GRU — Gated Recurrent Unit (Simplified Gating)

The GRU was proposed by Cho et al. (2014) as a simpler alternative to the LSTM. It achieves similar performance with fewer parameters by making two key simplifications:

1. Merge the cell state and hidden state into a single state vector h⟨t⟩
2. Merge the forget and input gates into a single update gate z⟨t⟩ (if you update, you automatically forget the old value)

GRU Equations — Step by Step

Update Gate (z⟨t⟩) — "How much of the old state to keep"

The update gate serves the roles of both the LSTM's forget gate and input gate. A value of z = 1 means "keep the old hidden state completely" (copy through), while z = 0 means "replace entirely with the new candidate".

Reset Gate (r⟨t⟩) — "How much of the old state to use for the candidate"

The reset gate controls how much of the previous hidden state is used to compute the new candidate. When r = 0, the model "resets" and acts as if reading the first word of a new sentence.

Candidate Hidden State (h̃⟨t⟩)

Notice the reset gate is applied inside the tanh — it selectively zeros out parts of the previous hidden state before computing the candidate.

Hidden State Update (h⟨t⟩)

This is a convex combination of the old state and the new candidate. The elegance: if z = 1, h⟨t⟩ = h⟨t−1⟩ (perfect copy, gradient flows through unchanged). If z = 0, h⟨t⟩ = h̃⟨t⟩ (complete reset).

Complete GRU Equations — Summary

GRU Parameter Count

With 3 sets of parameters (update, reset, candidate) instead of LSTM's 4:

For n = 256, m = 100: Total = 3(256²) + 3(256)(100) + 3(256) = 196,608 + 76,800 + 768 = 274,176 parameters — 25% fewer than LSTM.

14.4 LSTM vs GRU — When to Use Each

14.5 Bidirectional RNNs — Reading Forward AND Backward

Consider the NER (Named Entity Recognition) task on this Indian news headline:

"Sachin scored a century at Wankhede while Tendulkar Foundation donated ₹5 crore."

A forward-only RNN reading "Sachin" doesn't yet know what follows. Is "Sachin" a person's first name, a place, or a brand? Only when the model reads "scored a century" does it become clear this is a cricketer. And "Tendulkar Foundation" — is "Tendulkar" a person or an organization? Only the following word "Foundation" disambiguates it.

Architecture: Two RNNs, One Sequence

A Bidirectional RNN runs two separate RNNs on the same input:

• Forward RNN (→): Processes x⟨1⟩, x⟨2⟩, ..., x⟨T⟩ left-to-right, producing hidden states →a⟨1⟩, →a⟨2⟩, ..., →a⟨T⟩
• Backward RNN (←): Processes x⟨T⟩, x⟨T−1⟩, ..., x⟨1⟩ right-to-left, producing hidden states ←a⟨1⟩, ←a⟨2⟩, ..., ←a⟨T⟩

The prediction at each time step uses both past context (from forward RNN) and future context (from backward RNN):

14.6 Deep (Stacked) RNNs — Adding Depth to Sequence Models

Just as we stack convolutional layers in CNNs, we can stack multiple LSTM/GRU layers. The hidden state output of layer l becomes the input to layer l+1:

where a⟨t⟩_0 = x⟨t⟩ (the input embedding).

Practical guidelines for stacking:

• 2-3 layers is the sweet spot for most NLP tasks
• 4+ layers is rarely beneficial — diminishing returns + much slower training
• Google's Neural Machine Translation (GNMT) used 8 stacked LSTM layers with residual connections — but this was before Transformers took over
• Add dropout between layers (not within recurrent connections!) — typically 0.2-0.5

From-Scratch Code — LSTM Cell in NumPy

Let's implement a single LSTM cell forward pass using only NumPy. This computes one time step of the LSTM equations.

Python
import numpy as np

def sigmoid(x):
    """Numerically stable sigmoid."""
    return np.where(x >= 0,
                    1 / (1 + np.exp(-x)),
                    np.exp(x) / (1 + np.exp(x)))

def lstm_cell_forward(x_t, a_prev, c_prev, parameters):
    """
    Single LSTM cell forward pass.
    
    Arguments:
        x_t     -- input at time step t, shape (m, 1)
        a_prev  -- hidden state from previous step, shape (n, 1)
        c_prev  -- cell state from previous step, shape (n, 1)
        parameters -- dict containing:
            Wf, bf -- forget gate weights & bias
            Wi, bi -- input gate weights & bias
            Wc, bc -- cell candidate weights & bias
            Wo, bo -- output gate weights & bias
    
    Returns:
        a_next  -- next hidden state, shape (n, 1)
        c_next  -- next cell state, shape (n, 1)
        cache   -- values needed for backprop
    """
    # Extract parameters
    Wf = parameters["Wf"]  # shape (n, n+m)
    bf = parameters["bf"]  # shape (n, 1)
    Wi = parameters["Wi"]
    bi = parameters["bi"]
    Wc = parameters["Wc"]
    bc = parameters["bc"]
    Wo = parameters["Wo"]
    bo = parameters["bo"]
    
    # Step 1: Concatenate a_prev and x_t
    concat = np.vstack((a_prev, x_t))  # shape (n+m, 1)
    
    # Step 2: Forget gate — what to erase from cell state
    ft = sigmoid(Wf @ concat + bf)     # shape (n, 1)
    
    # Step 3: Input (update) gate — what new info to write
    it = sigmoid(Wi @ concat + bi)     # shape (n, 1)
    
    # Step 4: Cell candidate — proposed new memory
    c_tilde = np.tanh(Wc @ concat + bc)  # shape (n, 1)
    
    # Step 5: Cell state update (THE KEY EQUATION)
    c_next = ft * c_prev + it * c_tilde   # element-wise
    
    # Step 6: Output gate — what to reveal
    ot = sigmoid(Wo @ concat + bo)     # shape (n, 1)
    
    # Step 7: Hidden state — filtered cell state
    a_next = ot * np.tanh(c_next)      # shape (n, 1)
    
    # Cache for backpropagation
    cache = (a_next, c_next, a_prev, c_prev, ft, it,
             c_tilde, ot, x_t, parameters)
    
    return a_next, c_next, cache


def lstm_forward(x, a0, parameters):
    """
    Full LSTM forward pass over T time steps.
    
    Arguments:
        x  -- input sequence, shape (m, T)
        a0 -- initial hidden state, shape (n, 1)
        parameters -- dict of LSTM weights
    
    Returns:
        a_all -- all hidden states, shape (n, T)
        caches -- list of caches for backprop
    """
    n = a0.shape[0]
    m, T = x.shape
    
    # Initialize
    a_all = np.zeros((n, T))
    c_prev = np.zeros((n, 1))
    a_prev = a0
    caches = []
    
    for t in range(T):
        x_t = x[:, t].reshape(-1, 1)
        a_prev, c_prev, cache = lstm_cell_forward(
            x_t, a_prev, c_prev, parameters
        )
        a_all[:, t] = a_prev.flatten()
        caches.append(cache)
    
    return a_all, caches


# ─── Demo: Run LSTM on a toy sequence ───
np.random.seed(42)
n_hidden = 4   # hidden state size
n_input  = 3   # input feature size
T_steps  = 5   # sequence length

# Initialize parameters (Xavier-like)
scale = np.sqrt(2.0 / (n_hidden + n_input))
params = {}
for name in ["Wf", "Wi", "Wc", "Wo"]:
    params[name] = np.random.randn(n_hidden, n_hidden + n_input) * scale
for name in ["bf", "bi", "bc", "bo"]:
    params[name] = np.zeros((n_hidden, 1))
params["bf"] += 1.0   # Forget gate bias init = 1 (remember by default)

# Create toy input and initial hidden state
x_seq = np.random.randn(n_input, T_steps)
a_init = np.zeros((n_hidden, 1))

# Forward pass
hidden_states, caches = lstm_forward(x_seq, a_init, params)

print("Input shape:", x_seq.shape)
print("Hidden states shape:", hidden_states.shape)
print("\nHidden state at t=0:")
print(np.round(hidden_states[:, 0], 4))
print("\nHidden state at t=4:")
print(np.round(hidden_states[:, 4], 4))
print(f"\nTotal parameters: {sum(p.size for p in params.values()):,}")

Now let's also implement a GRU cell from scratch for comparison:

Python
def gru_cell_forward(x_t, h_prev, parameters):
    """
    Single GRU cell forward pass.
    
    Arguments:
        x_t    -- input at time step t, shape (m, 1)
        h_prev -- hidden state from previous step, shape (n, 1)
        parameters -- dict containing:
            Wz, bz -- update gate weights & bias
            Wr, br -- reset gate weights & bias
            Wh, bh -- candidate weights & bias
    
    Returns:
        h_next -- next hidden state, shape (n, 1)
    """
    Wz, bz = parameters["Wz"], parameters["bz"]
    Wr, br = parameters["Wr"], parameters["br"]
    Wh, bh = parameters["Wh"], parameters["bh"]
    
    # Concatenate h_prev and x_t
    concat = np.vstack((h_prev, x_t))   # (n+m, 1)
    
    # Update gate: how much to keep old state
    zt = sigmoid(Wz @ concat + bz)       # (n, 1)
    
    # Reset gate: how much old state to use in candidate
    rt = sigmoid(Wr @ concat + br)       # (n, 1)
    
    # Candidate hidden state (note: reset applied to h_prev)
    concat_reset = np.vstack((rt * h_prev, x_t))
    h_tilde = np.tanh(Wh @ concat_reset + bh)  # (n, 1)
    
    # Final hidden state: convex combination
    h_next = zt * h_prev + (1 - zt) * h_tilde
    
    return h_next

# Compare parameter counts
n, m = 256, 100
lstm_params = 4 * (n * (n + m) + n)
gru_params  = 3 * (n * (n + m) + n)
print(f"LSTM params (n={n}, m={m}): {lstm_params:,}")
print(f"GRU params  (n={n}, m={m}): {gru_params:,}")
print(f"GRU saves: {lstm_params - gru_params:,} params ({(lstm_params-gru_params)/lstm_params*100:.1f}%)")

Industry Code — TensorFlow / Keras

5A. NSE Nifty 50 Stock Price Prediction with LSTM

We build a model to predict the next-day closing price of the NSE Nifty 50 index using the past 60 days of prices. This is a classic many-to-one sequence problem.

Python
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau
from sklearn.preprocessing import MinMaxScaler
import matplotlib.pyplot as plt

# ─── 1. Load and Prepare Nifty 50 Data ───
# Download from: https://www.nseindia.com/market-data/live-equity-market
# Or use yfinance: pip install yfinance
import yfinance as yf

nifty = yf.download("^NSEI", start="2015-01-01", end="2024-12-31")
prices = nifty["Close"].values.reshape(-1, 1)

print(f"Dataset: {len(prices)} trading days")
print(f"Price range: ₹{prices.min():.0f} to ₹{prices.max():.0f}")

# ─── 2. Normalize prices to [0, 1] ───
scaler = MinMaxScaler(feature_range=(0, 1))
prices_scaled = scaler.fit_transform(prices)

# ─── 3. Create sequences: 60 days → predict day 61 ───
LOOKBACK = 60  # Use 60 days of history

def create_sequences(data, lookback):
    X, y = [], []
    for i in range(lookback, len(data)):
        X.append(data[i - lookback:i, 0])
        y.append(data[i, 0])
    return np.array(X), np.array(y)

X, y = create_sequences(prices_scaled, LOOKBACK)
X = X.reshape(X.shape[0], X.shape[1], 1)  # (samples, timesteps, features)

# Train-test split (80-20, chronological — NEVER shuffle time series!)
split = int(len(X) * 0.8)
X_train, X_test = X[:split], X[split:]
y_train, y_test = y[:split], y[split:]

print(f"Train: {X_train.shape}, Test: {X_test.shape}")

# ─── 4. Build Stacked LSTM Model ───
model = Sequential([
    # Layer 1: LSTM with 128 units, return sequences for stacking
    LSTM(128, return_sequences=True,
         input_shape=(LOOKBACK, 1)),
    Dropout(0.2),
    
    # Layer 2: LSTM with 64 units
    LSTM(64, return_sequences=False),
    Dropout(0.2),
    
    # Dense output: predict single price value
    Dense(32, activation="relu"),
    Dense(1)  # Linear activation for regression
])

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss="mse",
    metrics=["mae"]
)

model.summary()

# ─── 5. Train with Callbacks ───
callbacks = [
    EarlyStopping(monitor="val_loss", patience=10,
                  restore_best_weights=True),
    ReduceLROnPlateau(monitor="val_loss", factor=0.5,
                      patience=5, min_lr=1e-6)
]

history = model.fit(
    X_train, y_train,
    epochs=100,
    batch_size=32,
    validation_split=0.1,
    callbacks=callbacks,
    verbose=1
)

# ─── 6. Evaluate and Visualize ───
predictions_scaled = model.predict(X_test)
predictions = scaler.inverse_transform(predictions_scaled)
actual = scaler.inverse_transform(y_test.reshape(-1, 1))

# Metrics
mae = np.mean(np.abs(predictions - actual))
mape = np.mean(np.abs((actual - predictions) / actual)) * 100
print(f"\nTest MAE: ₹{mae:.2f}")
print(f"Test MAPE: {mape:.2f}%")

# Plot
plt.figure(figsize=(14, 5))
plt.plot(actual, label="Actual Nifty 50", color="#0f172a")
plt.plot(predictions, label="LSTM Prediction", color="#7c3aed", alpha=0.8)
plt.title("Nifty 50 Stock Price Prediction — LSTM")
plt.xlabel("Trading Days")
plt.ylabel("Price (₹)")
plt.legend()
plt.grid(alpha=0.3)
plt.tight_layout()
plt.savefig("nifty50_lstm_prediction.png", dpi=150)
plt.show()

5B. Bidirectional LSTM for Named Entity Recognition (Indian News)

We build a BiLSTM model to identify named entities (Person, Organization, Location) in Indian news text.

Python
import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import (
    Embedding, Bidirectional, LSTM, Dense, TimeDistributed, Dropout
)
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.utils import to_categorical

# ─── 1. Sample Indian News NER Dataset ───
# NER Tags: O=Outside, B-PER=Begin Person, I-PER=Inside Person,
# B-ORG=Begin Org, I-ORG=Inside Org, B-LOC=Begin Location

sentences = [
    ["Narendra", "Modi", "visited", "Varanasi", "yesterday"],
    ["TCS", "reported", "strong", "quarterly", "results"],
    ["Flipkart", "CEO", "Kalyan", "Krishnamurthy", "spoke",
     "at", "Bangalore", "tech", "summit"],
    ["HDFC", "Bank", "launched", "UPI", "services",
     "in", "Mumbai"],
    ["Sundar", "Pichai", "announced", "Google", "investment",
     "in", "India"],
]

labels = [
    ["B-PER", "I-PER", "O", "B-LOC", "O"],
    ["B-ORG", "O", "O", "O", "O"],
    ["B-ORG", "O", "B-PER", "I-PER", "O",
     "O", "B-LOC", "O", "O"],
    ["B-ORG", "I-ORG", "O", "B-ORG", "O",
     "O", "B-LOC"],
    ["B-PER", "I-PER", "O", "B-ORG", "O",
     "O", "B-LOC"],
]

# ─── 2. Build Vocabulary and Tag Mappings ───
words = sorted(set(w for s in sentences for w in s))
tags  = sorted(set(t for l in labels for t in l))

word2idx = {w: i+2 for i, w in enumerate(words)}  # 0=PAD, 1=UNK
word2idx["PAD"] = 0
word2idx["UNK"] = 1
tag2idx = {t: i for i, t in enumerate(tags)}
idx2tag = {i: t for t, i in tag2idx.items()}

n_words = len(word2idx)
n_tags  = len(tag2idx)
MAX_LEN = 15

# ─── 3. Encode and Pad Sequences ───
X = [[word2idx.get(w, 1) for w in s] for s in sentences]
y = [[tag2idx[t] for t in l] for l in labels]

X_pad = pad_sequences(X, maxlen=MAX_LEN, padding="post")
y_pad = pad_sequences(y, maxlen=MAX_LEN, padding="post",
                       value=tag2idx["O"])
y_cat = to_categorical(y_pad, num_classes=n_tags)

# ─── 4. Build BiLSTM Model ───
EMBED_DIM = 64
LSTM_UNITS = 128

model = Sequential([
    Embedding(input_dim=n_words, output_dim=EMBED_DIM,
              input_length=MAX_LEN, mask_zero=True),
    
    # Bidirectional LSTM: forward + backward = 256 dims
    Bidirectional(LSTM(LSTM_UNITS, return_sequences=True)),
    Dropout(0.3),
    
    # Second BiLSTM layer
    Bidirectional(LSTM(64, return_sequences=True)),
    Dropout(0.3),
    
    # TimeDistributed: apply Dense to each time step
    TimeDistributed(Dense(n_tags, activation="softmax"))
])

model.compile(
    optimizer="adam",
    loss="categorical_crossentropy",
    metrics=["accuracy"]
)

model.summary()

# ─── 5. Train (in production, use thousands of sentences) ───
model.fit(X_pad, y_cat, epochs=50, batch_size=2, verbose=0)

# ─── 6. Predict on a New Sentence ───
test_sentence = ["Ratan", "Tata", "founded", "Tata",
                 "Digital", "in", "Pune"]
test_encoded = [word2idx.get(w, 1) for w in test_sentence]
test_padded = pad_sequences([test_encoded], maxlen=MAX_LEN,
                             padding="post")

pred = model.predict(test_padded, verbose=0)
pred_tags = [idx2tag[np.argmax(p)] for p in pred[0][:len(test_sentence)]]

print("\nNER Predictions:")
print("-" * 40)
for word, tag in zip(test_sentence, pred_tags):
    marker = "  ◄" if tag != "O" else ""
    print(f"  {word:20s} → {tag}{marker}")

Visual Diagrams

6A. LSTM Cell — Complete Architecture

6B. GRU Cell — Simplified Architecture

6C. LSTM vs GRU — Side by Side

6D. Unrolled Bidirectional LSTM

Worked Example — Tracing Through an LSTM Cell by Hand

Let's trace through one time step of an LSTM cell with concrete numbers. We'll use tiny dimensions (n=2, m=2) to make the math tractable.

Setup

Hidden size n = 2, Input size m = 2. The concatenated vector [a⟨t−1⟩, x⟨t⟩] has size 4.

Step 1: Forget Gate

W_f · [a, x] = [0.2(0.5) + 0.1(−0.3) + 0.3(1.0) + (−0.1)(0.5), 0.1(0.5) + 0.4(−0.3) + (−0.2)(1.0) + 0.2(0.5)]

= [0.1 − 0.03 + 0.3 − 0.05, 0.05 − 0.12 − 0.2 + 0.1] = [0.32, −0.17]

f⟨t⟩ = σ([0.32, −0.17] + [1.0, 1.0]) = σ([1.32, 0.83])

✅ Both values are close to 1 → the LSTM remembers most of the old cell state (because we initialized b_f = 1).

Step 2: Input Gate

W_i · [a, x] = [0.3(0.5) + (−0.1)(−0.3) + 0.2(1.0) + 0.1(0.5), (−0.2)(0.5) + 0.3(−0.3) + 0.1(1.0) + 0.4(0.5)]

= [0.15 + 0.03 + 0.2 + 0.05, −0.1 − 0.09 + 0.1 + 0.2] = [0.43, 0.11]

Step 3: Cell Candidate

W_c · [a, x] = [0.1(0.5) + 0.2(−0.3) + 0.5(1.0) + (−0.3)(0.5), 0.4(0.5) + (−0.1)(−0.3) + 0.3(1.0) + 0.2(0.5)]

= [0.05 − 0.06 + 0.5 − 0.15, 0.2 + 0.03 + 0.3 + 0.1] = [0.34, 0.63]

Step 4: Cell State Update (THE KEY STEP)

📊 Analysis: Dimension 1 changed from 0.8 to 0.830 (slight increase — mostly remembered + small new info). Dimension 2 changed from 1.2 to 1.129 (slight decrease — forgot ~30% of old value, added new info).

Step 5: Output Gate

W_o · [a, x] = [−0.1(0.5) + 0.3(−0.3) + 0.2(1.0) + 0.1(0.5), 0.2(0.5) + 0.1(−0.3) + (−0.3)(1.0) + 0.5(0.5)]

= [−0.05 − 0.09 + 0.2 + 0.05, 0.1 − 0.03 − 0.3 + 0.25] = [0.11, 0.02]

Step 6: Hidden State

Case Study — HDFC Bank: LSTM-Powered Fraud Detection

Common Mistakes & Misconceptions

Comparison Table — RNN Architectures

When to Choose What — Decision Flowchart

Exercises

Section A — Multiple Choice Questions (10)

Section B — Short Answer Questions (5)

Section C — Long Answer Questions (3)

Section D — Programming Assignments (2)

Sentence: "Modi ji ne Varanasi mein rally ki"
Tags:      B-PER O O  B-LOC   O    O    O

Sentence: "Flipkart ka Big Billion Days sale start"
Tags:      B-ORG  O  O   O       O    O    O

Section E — Mini-Project

Chapter Summary

References

Foundational Papers

1. Hochreiter, S. & Schmidhuber, J. (1997). "Long Short-Term Memory." Neural Computation, 9(8), 1735–1780. — The original LSTM paper introducing the cell state and gating mechanism.
2. Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). "Learning to Forget: Continual Prediction with LSTM." Neural Computation, 12(10), 2451–2471. — Added the forget gate (not in the original 1997 paper!).
3. Cho, K. et al. (2014). "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation." EMNLP 2014. — The paper introducing GRU.
4. Chung, J. et al. (2014). "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling." NIPS 2014 Workshop. — Comprehensive LSTM vs GRU comparison.
5. Jozefowicz, R., Zaremba, W., & Sutskever, I. (2015). "An Empirical Exploration of Recurrent Network Architectures." ICML 2015. — Showed forget gate bias initialization to 1.0 is critical.

Architecture Variants

6. Schuster, M. & Paliwal, K. K. (1997). "Bidirectional Recurrent Neural Networks." IEEE Transactions on Signal Processing, 45(11). — The original bidirectional RNN paper.
7. Graves, A. (2013). "Generating Sequences With Recurrent Neural Networks." arXiv:1308.0850. — Handwriting generation with stacked LSTMs.
8. Wu, Y. et al. (2016). "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation." arXiv:1609.08144. — GNMT with 8-layer stacked LSTMs and residual connections.
9. Gal, Y. & Ghahramani, Z. (2016). "A Theoretically Grounded Application of Dropout to Recurrent Neural Networks." NIPS 2016. — Variational/recurrent dropout for LSTMs.

Textbooks

10. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Chapter 10: Sequence Modeling — Detailed treatment of LSTM and GRU architectures.
11. Chollet, F. (2021). Deep Learning with Python, 2nd Edition. Manning. Chapter 10 — Practical LSTM/GRU implementations in Keras.
12. Jurafsky, D. & Martin, J. H. (2023). Speech and Language Processing, 3rd Edition (Draft). Chapter 9 — LSTMs in NLP context.

Indian Industry Context

13. HDFC Bank Annual Report (2023-24) — Sections on digital banking technology, AI/ML fraud detection deployment statistics.
14. RBI Circular on Digital Payment Security Controls (2022) — Mandate for AI/ML-based fraud detection in Indian banking, driving LSTM adoption.
15. NASSCOM AI in BFSI Report (2023) — Survey of AI/ML adoption in Indian banking and financial services, including sequence modeling for fraud detection and credit scoring.