Chapter 16: Autoencoders and Representation Learning

Bloom's Taxonomy Map for This Chapter

Learning Objectives

After completing this chapter, you will be able to:

1. Define the autoencoder framework — encoder function f(x), latent code z, decoder function g(z) — and explain why training it as an identity function is not trivial
2. Distinguish between undercomplete (bottleneck < input dim) and overcomplete (bottleneck ≥ input dim) autoencoders, and explain when each is useful
3. Derive the sparse autoencoder loss with KL-divergence penalty and implement it in both NumPy and PyTorch
4. Explain why denoising autoencoders learn more robust features than vanilla AEs, and how contractive autoencoders achieve the same via Jacobian penalty
5. Derive the ELBO (Evidence Lower Bound) for Variational Autoencoders from first principles — including the reparameterization trick — and implement a complete VAE
6. Apply autoencoders to real-world tasks: anomaly detection (Paytm UPI fraud), audio feature learning (Spotify), dimensionality reduction, and denoising
7. Compare autoencoders with PCA — proving that a linear AE with MSE loss recovers PCA projections
8. Visualize and interpret 2D latent spaces, perform latent space interpolation, and generate new samples from a trained VAE

Opening Hook

The Intuition First

The Suitcase Analogy

Imagine you're packing for a two-week trip, but your airline only allows a tiny carry-on bag. You can't bring everything — you must decide what's essential. You fold clothes tightly, choose versatile items, and leave behind the non-essentials.

Now imagine your friend at the destination has to reconstruct your entire wardrobe from what arrives in that tiny bag. If they can do it well — if the essentials you packed are enough to approximate everything you needed — then you've learned an excellent compression.

That's exactly what an autoencoder does:

• Encoder = You packing the suitcase (compress 784 MNIST pixels → 32 numbers)
• Bottleneck = The tiny carry-on bag (the 32-dimensional latent code z)
• Decoder = Your friend unpacking and reconstructing (32 numbers → 784 pixels)
• Training signal = How different is the reconstruction from the original?

The "Aha" Question

Why would you train a network to output its own input? That sounds trivially useless — the identity function does this perfectly. The magic is in the constraint: by forcing the data through a narrow bottleneck, you prevent the network from learning the identity and instead force it to learn which features matter most. The bottleneck IS the learned representation.

The Autoencoder Architecture

Formal Definition

An autoencoder consists of two functions:

where:

• x ∈ ℝn is the input (e.g., a 784-dimensional flattened MNIST image)
• z ∈ ℝd is the latent code (bottleneck representation), with d < n for undercomplete AEs
• x̂ ∈ ℝn is the reconstruction
• θ, φ are the encoder and decoder parameters (weights & biases)

You train by minimizing the reconstruction loss over your dataset:

A Concrete Architecture

For MNIST (28×28 = 784 pixels), a simple autoencoder might look like:

Undercomplete vs Overcomplete Autoencoders

Undercomplete AE (d < n)

When the latent dimension d is smaller than the input dimension n, the autoencoder is undercomplete. It must learn to compress — there simply isn't enough capacity in the bottleneck to memorize the input.

This is the classic, intuitive case. Think of it like summarizing a novel in a tweet: you're forced to extract the essential meaning.

Overcomplete AE (d ≥ n)

What if the latent dimension is larger than or equal to the input? Now the network has more capacity than needed — it can trivially learn the identity function by copying each input dimension to a latent dimension.

This sounds useless, but with the right regularization, overcomplete autoencoders learn powerful features:

• Sparse AE: Penalize activations so most latent units are inactive → learns selective features
• Denoising AE: Corrupt input, reconstruct clean version → learns robust features
• Contractive AE: Penalize the Jacobian of the encoder → learns locally invariant features

Sparse Autoencoders

The Intuition

Imagine a classroom of 500 students (neurons), but only 20 should raise their hands for any given question. Each student specializes in recognizing something specific. When you show a cat picture, only the "whiskers," "pointy ears," and "fur texture" students activate — the rest stay silent. That's sparsity.

Mathematical Foundation

For each hidden neuron j, define the average activation over the training set:

where aⱼ(xᵢ) is the activation of neuron j for input xᵢ. We want ρ̂ⱼ ≈ ρ, where ρ is a small target (e.g., 0.05).

We penalize deviation from the target using KL divergence:

The total sparse autoencoder loss becomes:

where β controls the strength of the sparsity penalty (typically β = 3, ρ = 0.05).

Denoising Autoencoders (DAE)

The Key Idea

Instead of training the autoencoder on clean data, you corrupt the input first — add Gaussian noise, randomly zero-out pixels (masking noise), or apply salt-and-pepper noise — then train the decoder to reconstruct the clean original.

Common Corruption Methods

Why Does This Work?

By corrupting the input, you force the autoencoder to learn the statistical structure of the data distribution — not just memorize inputs. The network must figure out: "Given that pixel (3,7) is corrupted, what should it be, based on the surrounding pixels?" This requires learning the data manifold.

Contractive Autoencoders (CAE)

The Intuition

Think of the encoder as a mapping from input space to latent space. A contractive autoencoder says: "Small changes in the input should produce even smaller changes in the latent code." In other words, the encoding should be locally robust — it shouldn't jump around wildly when you slightly perturb the input.

The Jacobian Penalty

Formally, we penalize the Frobenius norm of the Jacobian of the encoder:

The Jacobian matrix J ∈ ℝd×n has entry Jᵢⱼ = ∂zᵢ/∂xⱼ — how much does latent dimension i change when input dimension j changes. Penalizing this forces the encoder to be insensitive to small input variations, learning only the directions that truly matter.

Variational Autoencoders (VAE)

This is the crown jewel of this chapter. Buckle up — we'll derive everything from first principles.

The Problem with Regular Autoencoders

A vanilla AE learns a deterministic mapping: each input x maps to exactly one point z in latent space. The latent space has no structure — points can be scattered arbitrarily. If you sample a random point z from this latent space and decode it, you'll likely get garbage. Regular AEs compress but cannot generate.

The VAE Idea: Make the Latent Space Probabilistic

Instead of encoding x to a single point z, we encode it to a probability distribution — specifically, a Gaussian N(μ, σ²). Then we sample z from this distribution. The decoder must be able to reconstruct x from any sample drawn from the distribution, not just one specific point.

The ELBO Derivation (From Scratch)

The Two Terms Explained

Term 1: Reconstruction Loss — 𝔼_q[log p(x|z)]

This is the expected log-likelihood of the data given the latent code. In practice:

• For binary/normalized data: Binary Cross-Entropy between x and x̂
• For continuous data: MSE (equivalent to Gaussian p(x|z) with fixed variance)

Term 2: KL Divergence — KL(q(z|x) ‖ p(z))

This regularizes the latent space, pushing q(z|x) = N(μ, σ²) toward the prior p(z) = N(0, I). It has a closed-form solution for two Gaussians:

The Reparameterization Trick

Here's a subtle problem: how do you backpropagate through a sampling operation? If z ~ N(μ, σ²), the sampling is stochastic — gradients can't flow through random number generators.

Solution: Instead of sampling z directly, reparameterize:

Now the randomness is in ε (which has no learnable parameters), and z is a deterministic function of μ and σ. Gradients flow through μ and σ just fine!

Autoencoders vs PCA

The Theorem: Linear AE = PCA

Here's a beautiful result. Consider a single-layer autoencoder with no activation function (linear encoder and decoder) trained with MSE loss:

The optimal weights W span the same subspace as the top-d principal components of the data! The latent code z is (a rotation of) the PCA projection.

When AEs Beat PCA

Anomaly Detection with Autoencoders

The Core Idea

Train an autoencoder on normal data only. The AE learns to compress and reconstruct normal patterns well. When an anomalous input arrives, the AE fails to reconstruct it — the reconstruction error spikes. You flag any input with reconstruction error above a threshold as anomalous.

🇮🇳 Indian Industry: Paytm UPI Anomaly Detection

🇺🇸 US/Global Industry: Spotify Audio Feature Learning

Worked Examples

Example 1: By-Hand Forward Pass (Vanilla AE)

Example 2: Computing VAE KL Divergence

Example 3: Anomaly Detection Threshold (Indian Industry)

Python Implementation: From Scratch (NumPy)

Vanilla Autoencoder — NumPy

Python · NumPy
import numpy as np

class VanillaAutoencoder:
    """Vanilla AE: 784 → 128 → 32 → 128 → 784"""
    def __init__(self, input_dim=784, hidden_dim=128, latent_dim=32, lr=0.001):
        self.lr = lr
        # Xavier initialization
        self.W1 = np.random.randn(input_dim, hidden_dim) * np.sqrt(2.0 / input_dim)
        self.b1 = np.zeros(hidden_dim)
        self.W2 = np.random.randn(hidden_dim, latent_dim) * np.sqrt(2.0 / hidden_dim)
        self.b2 = np.zeros(latent_dim)
        self.W3 = np.random.randn(latent_dim, hidden_dim) * np.sqrt(2.0 / latent_dim)
        self.b3 = np.zeros(hidden_dim)
        self.W4 = np.random.randn(hidden_dim, input_dim) * np.sqrt(2.0 / hidden_dim)
        self.b4 = np.zeros(input_dim)

    def relu(self, x):
        return np.maximum(0, x)

    def relu_grad(self, x):
        return (x > 0).astype(float)

    def sigmoid(self, x):
        return 1.0 / (1.0 + np.exp(-np.clip(x, -500, 500)))

    def forward(self, x):
        # Encoder
        self.z1 = x @ self.W1 + self.b1
        self.a1 = self.relu(self.z1)                # (batch, 128)
        self.z2 = self.a1 @ self.W2 + self.b2
        self.latent = self.z2                        # (batch, 32) — linear bottleneck
        # Decoder
        self.z3 = self.latent @ self.W3 + self.b3
        self.a3 = self.relu(self.z3)                # (batch, 128)
        self.z4 = self.a3 @ self.W4 + self.b4
        self.output = self.sigmoid(self.z4)          # (batch, 784)
        return self.output

    def compute_loss(self, x, x_hat):
        # Binary cross-entropy
        eps = 1e-8
        bce = -np.mean(x * np.log(x_hat + eps) + (1 - x) * np.log(1 - x_hat + eps))
        return bce

    def backward(self, x):
        batch_size = x.shape[0]
        # Output gradient (BCE derivative with sigmoid)
        dz4 = (self.output - x) / batch_size          # (batch, 784)
        dW4 = self.a3.T @ dz4
        db4 = dz4.sum(axis=0)
        da3 = dz4 @ self.W4.T
        dz3 = da3 * self.relu_grad(self.z3)
        dW3 = self.latent.T @ dz3
        db3 = dz3.sum(axis=0)
        dlatent = dz3 @ self.W3.T
        # Latent layer is linear, so dz2 = dlatent
        dz2 = dlatent
        dW2 = self.a1.T @ dz2
        db2 = dz2.sum(axis=0)
        da1 = dz2 @ self.W2.T
        dz1 = da1 * self.relu_grad(self.z1)
        dW1 = x.T @ dz1
        db1 = dz1.sum(axis=0)
        # Update weights
        for W, dW in [(self.W1, dW1), (self.W2, dW2),
                        (self.W3, dW3), (self.W4, dW4)]:
            W -= self.lr * dW
        for b, db in [(self.b1, db1), (self.b2, db2),
                        (self.b3, db3), (self.b4, db4)]:
            b -= self.lr * db

    def train_step(self, x_batch):
        x_hat = self.forward(x_batch)
        loss = self.compute_loss(x_batch, x_hat)
        self.backward(x_batch)
        return loss

# ── Usage ──
# from sklearn.datasets import fetch_openml
# mnist = fetch_openml('mnist_784', version=1)
# X = mnist.data.values / 255.0  # normalize to [0,1]
# ae = VanillaAutoencoder()
# for epoch in range(50):
#     for i in range(0, len(X), 128):
#         loss = ae.train_step(X[i:i+128])
#     print(f"Epoch {epoch}: loss={loss:.4f}")

Variational Autoencoder — NumPy

Python · NumPy
class VAE_NumPy:
    """Variational AE: 784 → 256 → (μ,logvar) → z(2-dim) → 256 → 784"""
    def __init__(self, input_dim=784, hidden=256, latent=2, lr=0.001):
        self.lr = lr
        self.latent = latent
        # Encoder: input → hidden → (mu, logvar)
        self.W1 = np.random.randn(input_dim, hidden) * np.sqrt(2.0/input_dim)
        self.b1 = np.zeros(hidden)
        self.W_mu = np.random.randn(hidden, latent) * np.sqrt(2.0/hidden)
        self.b_mu = np.zeros(latent)
        self.W_lv = np.random.randn(hidden, latent) * np.sqrt(2.0/hidden)
        self.b_lv = np.zeros(latent)
        # Decoder: z → hidden → output
        self.W3 = np.random.randn(latent, hidden) * np.sqrt(2.0/latent)
        self.b3 = np.zeros(hidden)
        self.W4 = np.random.randn(hidden, input_dim) * np.sqrt(2.0/hidden)
        self.b4 = np.zeros(input_dim)

    def relu(self, x): return np.maximum(0, x)
    def relu_d(self, x): return (x > 0).astype(float)
    def sigmoid(self, x): return 1/(1+np.exp(-np.clip(x,-500,500)))

    def forward(self, x):
        B = x.shape[0]
        # Encode
        self.h1_pre = x @ self.W1 + self.b1
        self.h1 = self.relu(self.h1_pre)
        self.mu = self.h1 @ self.W_mu + self.b_mu       # (B, latent)
        self.logvar = self.h1 @ self.W_lv + self.b_lv    # (B, latent)
        # Reparameterize: z = mu + exp(0.5*logvar) * eps
        self.eps = np.random.randn(B, self.latent)
        self.std = np.exp(0.5 * self.logvar)
        self.z = self.mu + self.std * self.eps           # (B, latent)
        # Decode
        self.h3_pre = self.z @ self.W3 + self.b3
        self.h3 = self.relu(self.h3_pre)
        self.out_pre = self.h3 @ self.W4 + self.b4
        self.x_hat = self.sigmoid(self.out_pre)          # (B, 784)
        return self.x_hat

    def loss(self, x):
        eps = 1e-8
        # Reconstruction: BCE
        recon = -np.sum(x*np.log(self.x_hat+eps) + (1-x)*np.log(1-self.x_hat+eps)) / x.shape[0]
        # KL: -0.5 * sum(1 + logvar - mu^2 - exp(logvar))
        kl = -0.5 * np.sum(1 + self.logvar - self.mu**2 - np.exp(self.logvar)) / x.shape[0]
        return recon + kl, recon, kl

    def backward(self, x):
        B = x.shape[0]
        # ── Decoder gradients ──
        d_out = (self.x_hat - x) / B                    # BCE+sigmoid shortcut
        dW4 = self.h3.T @ d_out
        db4 = d_out.sum(0)
        dh3 = d_out @ self.W4.T * self.relu_d(self.h3_pre)
        dW3 = self.z.T @ dh3
        db3 = dh3.sum(0)
        dz = dh3 @ self.W3.T                            # (B, latent)
        # ── Reparameterization gradients ──
        d_mu = dz + (self.mu) / B                        # recon + KL grad w.r.t μ
        d_logvar = dz * 0.5 * self.std * self.eps \
                   + 0.5 * (np.exp(self.logvar) - 1) / B  # KL grad w.r.t logvar
        # ── Encoder gradients ──
        dW_mu = self.h1.T @ d_mu
        db_mu = d_mu.sum(0)
        dW_lv = self.h1.T @ d_logvar
        db_lv = d_logvar.sum(0)
        dh1 = (d_mu @ self.W_mu.T + d_logvar @ self.W_lv.T) * self.relu_d(self.h1_pre)
        dW1 = x.T @ dh1
        db1 = dh1.sum(0)
        # ── Update all weights ──
        for p, g in [(self.W1,dW1),(self.b1,db1),(self.W_mu,dW_mu),(self.b_mu,db_mu),
                       (self.W_lv,dW_lv),(self.b_lv,db_lv),(self.W3,dW3),(self.b3,db3),
                       (self.W4,dW4),(self.b4,db4)]:
            p -= self.lr * g

    def train_step(self, x):
        self.forward(x)
        total, recon, kl = self.loss(x)
        self.backward(x)
        return total, recon, kl

# Usage: vae = VAE_NumPy(latent=2); vae.train_step(batch)

# BUGGY CODE — find the error!
kl_loss = -0.5 * np.sum(1 + self.logvar - self.mu**2 - self.logvar**2)

PyTorch Implementations

Vanilla Autoencoder — PyTorch

Python · PyTorch
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

class Autoencoder(nn.Module):
    def __init__(self, latent_dim=32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(784, 256), nn.ReLU(),
            nn.Linear(256, 128), nn.ReLU(),
            nn.Linear(128, latent_dim)      # no activation → linear bottleneck
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 128), nn.ReLU(),
            nn.Linear(128, 256), nn.ReLU(),
            nn.Linear(256, 784), nn.Sigmoid()
        )

    def forward(self, x):
        z = self.encoder(x)
        x_hat = self.decoder(z)
        return x_hat, z

# ── Training Loop ──
transform = transforms.Compose([transforms.ToTensor(), transforms.Lambda(lambda x: x.view(-1))])
train_data = datasets.MNIST('./data', train=True, download=True, transform=transform)
loader = DataLoader(train_data, batch_size=128, shuffle=True)

model = Autoencoder(latent_dim=32)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.BCELoss(reduction='mean')

for epoch in range(20):
    total_loss = 0
    for x, _ in loader:
        x_hat, z = model(x)
        loss = criterion(x_hat, x)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
    print(f"Epoch {epoch+1}: loss={total_loss/len(loader):.4f}")

Variational Autoencoder — PyTorch

Python · PyTorch
class VAE(nn.Module):
    def __init__(self, latent_dim=2):
        super().__init__()
        # Encoder: shared layers + two heads (mu, logvar)
        self.encoder = nn.Sequential(
            nn.Linear(784, 512), nn.ReLU(),
            nn.Linear(512, 256), nn.ReLU(),
        )
        self.fc_mu     = nn.Linear(256, latent_dim)
        self.fc_logvar = nn.Linear(256, latent_dim)
        # Decoder
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 256), nn.ReLU(),
            nn.Linear(256, 512), nn.ReLU(),
            nn.Linear(512, 784), nn.Sigmoid(),
        )

    def encode(self, x):
        h = self.encoder(x)
        return self.fc_mu(h), self.fc_logvar(h)

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)           # ε ~ N(0, I)
        return mu + std * eps                  # z = μ + σ⊙ε

    def decode(self, z):
        return self.decoder(z)

    def forward(self, x):
        mu, logvar = self.encode(x)
        z = self.reparameterize(mu, logvar)
        x_hat = self.decode(z)
        return x_hat, mu, logvar, z

def vae_loss(x, x_hat, mu, logvar):
    # Reconstruction: BCE
    recon = nn.functional.binary_cross_entropy(x_hat, x, reduction='sum')
    # KL divergence: -0.5 * Σ(1 + log(σ²) - μ² - σ²)
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return (recon + kl) / x.size(0), recon / x.size(0), kl / x.size(0)

# ── Training Loop ──
vae = VAE(latent_dim=2)
opt = optim.Adam(vae.parameters(), lr=1e-3)

for epoch in range(30):
    for x, _ in loader:
        x_hat, mu, logvar, z = vae(x)
        loss, recon, kl = vae_loss(x, x_hat, mu, logvar)
        opt.zero_grad()
        loss.backward()
        opt.step()
    print(f"Epoch {epoch+1}: loss={loss:.1f}  recon={recon:.1f}  kl={kl:.1f}")

Latent Space Visualization & Generation

Python · Visualization
import matplotlib.pyplot as plt

# ── 1. Visualize latent space (color by digit) ──
vae.eval()
all_z, all_y = [], []
with torch.no_grad():
    for x, y in loader:
        _, mu, _, _ = vae(x)
        all_z.append(mu.numpy())
        all_y.append(y.numpy())

Z = np.concatenate(all_z)
Y = np.concatenate(all_y)

plt.figure(figsize=(10, 8))
for digit in range(10):
    mask = Y == digit
    plt.scatter(Z[mask, 0], Z[mask, 1], s=2, alpha=0.5, label=str(digit))
plt.legend(); plt.title("VAE Latent Space (2D)")
plt.xlabel("z₁"); plt.ylabel("z₂")
plt.savefig("vae_latent_space.png", dpi=150)
plt.show()

# ── 2. Generate new digits by sampling from prior ──
n = 15
grid = np.linspace(-3, 3, n)
fig, axes = plt.subplots(n, n, figsize=(12, 12))
for i, yi in enumerate(grid):
    for j, xi in enumerate(grid):
        z = torch.tensor([[xi, yi]], dtype=torch.float32)
        with torch.no_grad():
            img = vae.decode(z).view(28, 28).numpy()
        axes[i][j].imshow(img, cmap='gray')
        axes[i][j].axis('off')
plt.suptitle("Generated Digits: Traversing 2D Latent Space")
plt.tight_layout()
plt.savefig("vae_generated_grid.png", dpi=150)
plt.show()

Visual Diagrams

Diagram 1: The Autoencoder Zoo

Diagram 2: VAE Loss Landscape

Diagram 3: AE vs PCA — Data Manifolds

Common Misconceptions

GATE / Exam Corner

Previous Year Questions & Patterns

Exam-Ready Formula Sheet

GATE Prediction Table (2025–2027)

Interview Prep

Conceptual Questions

Coding Questions

def vae_loss_fn(x, x_hat, mu, logvar):
    # Reconstruction: sum over features, mean over batch
    recon = F.binary_cross_entropy(x_hat, x, reduction='sum')
    # KL: closed-form for diagonal Gaussian vs N(0,I)
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return (recon + kl) / x.size(0)

def reparameterize(mu, logvar):
    std = torch.exp(0.5 * logvar)  # σ = exp(½ log σ²)
    eps = torch.randn_like(std)     # ε ~ N(0, I)
    return mu + std * eps            # z = μ + σ⊙ε

Case Study Questions

Hands-On Lab: MNIST VAE with Latent Space Exploration

Lab Objective

Build a complete VAE pipeline: train on MNIST, visualize the 2D latent space, generate new digits by sampling, and interpolate between digits in latent space.

Lab Steps

Grading Rubric

Exercises

Section A: Conceptual Questions (5)

Section B: Mathematical Questions (8)

Section C: Coding Questions (4)

Section D: Critical Thinking (3)

★ Starred Research Questions (2)

Connections

Chapter Summary

Further Reading

🇮🇳 Indian Resources

• NPTEL: "Deep Learning" by Prof. Mitesh Khapra (IIT Madras) — Lectures 35-38 cover autoencoders and VAEs with excellent mathematical rigor
• NPTEL: "Introduction to Machine Learning" by Prof. Sudeshna Sarkar (IIT Kharagpur) — PCA and dimensionality reduction context
• GATE Preparation: "Deep Learning" by Ian Goodfellow — Chapter 14 (Autoencoders) is the gold standard reference
• Padhai (One Fourth Labs): Free VAE tutorial series with Python implementation walkthroughs

🌐 Global Resources

• Original VAE Paper: "Auto-Encoding Variational Bayes" — Kingma & Welling (2014). The foundational paper. Read the derivation alongside our Section 16.6
• Tutorial: "An Introduction to Variational Autoencoders" — Kingma & Welling (2019). A comprehensive 50-page tutorial
• Distill.pub: "Understanding the Variational Lower Bound" — Interactive visualization of the ELBO
• 3Blue1Brown: "But what is a neural network?" — Foundation for understanding AE architectures
• Lilian Weng's Blog: "From Autoencoder to Beta-VAE" — Outstanding survey of all AE variants
• Paper: "Masked Autoencoders Are Scalable Vision Learners" — He et al. (2022). Modern connection to self-supervised learning
• Paper: "Neural Discrete Representation Learning" (VQ-VAE) — van den Oord et al. (2017)
• Goodfellow et al.: "Deep Learning" textbook, Chapter 14 (free online at deeplearningbook.org)