Chapter 8: Activation Functions — Adding Non-Linearity

Bloom's Taxonomy Map for This Chapter

Learning Objectives

By the end of this chapter, you will be able to:

• Prove mathematically that a neural network with only linear activations is equivalent to a single linear transformation, regardless of depth
• State the formula, derivative, output range, and computational cost for each of the 8 activation functions covered
• Explain the vanishing gradient problem in sigmoid and tanh, and why ReLU largely solved it
• Diagnose the dying ReLU problem — identify when neurons die, detect it in training logs, and apply fixes
• Compare GELU and Swish with ReLU, and explain why modern Transformer architectures (BERT, GPT) prefer GELU
• Derive the softmax function from a log-linear model and compute its Jacobian matrix
• Implement all activation functions and their derivatives from scratch in NumPy, and verify against PyTorch
• Select the right activation function for any given task using a systematic decision tree

Opening Hook

The Intuition First

The Valve Analogy

Imagine you're building a water distribution network for a city. You have pipes (weights) connecting various junctions (neurons), and water (data) flows through. If every junction is just a straight-through connection — no valves, no gates — then no matter how complex your pipe network is, the relationship between water in and water out is always linear. Add more pipes? Still linear. Make the network deeper? Still linear.

An activation function is like putting a valve at each junction. The valve can:

• Block flow entirely (like ReLU zeroing out negatives)
• Regulate flow (like sigmoid squashing it between 0 and 1)
• Amplify selectively (like ELU boosting small negative signals)

With valves, suddenly your network can create incredibly complex flow patterns — eddies, branches, feedback loops — that a straight pipe network never could.

The "Aha" Question

🤔 If ReLU is just max(0, x) — a function you could explain to a 10-year-old — why did it take until 2012 for the deep learning community to embrace it? And why did Google Brain spend years searching for something better?

By the end of this chapter, you'll not only understand the answer, but you'll be able to derive why certain activations work better for certain architectures — and make that choice yourself.

Mathematical Foundation: Why Non-Linearity is Essential

The Collapse Theorem: Stacked Linear Layers = Single Linear Layer

What Non-Linearity Buys You

The Universal Approximation Theorem (Cybenko, 1989; Hornik, 1991) states: a neural network with a single hidden layer and a non-linear activation function can approximate any continuous function to arbitrary accuracy, given enough hidden units.

The key phrase is "non-linear". Without it, you're stuck approximating only linear functions — planes in 2D, hyperplanes in higher dimensions. With it, you can learn spirals, circles, XOR, and anything else.

Activation 1: Sigmoid — The Classic S-Curve

Activation 2: Tanh — Zero-Centered Sigmoid

Activation 3: ReLU — The Game Changer

Activation 4: Leaky ReLU — Fixing the Dead Neuron Problem

Activation 5: ELU — Exponential Linear Unit

Activation 6: GELU — The Transformer's Choice

Activation 7: Swish / SiLU — The Neural Architecture Search Discovery

Activation 8: Softmax — Multi-Class Output

Activation Selection Guide — Decision Tree

The Master Comparison Table

Decision Tree: Which Activation to Choose?

Dying ReLU Investigation

What Is Dying ReLU?

A "dead" ReLU neuron is one where Wx + b < 0 for every single training example. Since ReLU(negative) = 0 and ReLU'(negative) = 0, the neuron outputs zero, receives zero gradient, and its weights never update. It's permanently stuck.

When Does It Happen?

1. Large learning rate: A big gradient update can push weights to a region where the neuron becomes negative for all inputs. Think of it as the neuron "jumping off a cliff."
2. Bad initialization: If weights are initialized too large (or with large negative bias), neurons start dead.
3. Input distribution shift: If the data distribution changes during training, previously active neurons can die.

How to Detect Dead Neurons

Python
# After a forward pass, check what fraction of neurons are dead
def check_dead_neurons(activations):
    """activations: dict of layer_name -> activation tensor"""
    for name, act in activations.items():
        # A neuron is "dead" if it outputs 0 for ALL examples in the batch
        dead_mask = (act == 0).all(axis=0)  # per-neuron check
        dead_frac = dead_mask.mean()
        print(f"{name}: {dead_frac*100:.1f}% neurons dead")
        if dead_frac > 0.5:
            print(f"  ⚠️ WARNING: More than 50% dead in {name}!")

# Healthy: 0-10% dead. Concerning: 10-30%. Critical: >50%

How to Fix Dying ReLU

Layer 1: 48.2% neurons dead
Layer 2: 67.1% neurons dead
Layer 3: 85.4% neurons dead
Layer 4: 97.3% neurons dead
Layer 5: 99.8% neurons dead

Worked Examples

Example 1: By-Hand Computation — All Activations for z = −2, 0, 2

Example 2: Gradient Flow — 5-Layer Network Comparison

Setup: A 5-layer network. Let's trace how a gradient signal of 1.0 at the output gets attenuated as it flows backward.

With sigmoid, the gradient reaching Layer 1 is only 0.39% of its original value! This is the vanishing gradient problem. With ReLU, gradients pass through unchanged (as long as the neuron is active). Note: tanh's best case is 1.0, but in practice tanh'(z) < 1 for z ≠ 0, so it also vanishes — just slower than sigmoid.

Example 3: Softmax Computation

Python Implementation — From Scratch (NumPy)

All 8 Activations + Derivatives

Python — NumPy
import numpy as np

# ═══════════════════════════════════════
# 1. SIGMOID
# ═══════════════════════════════════════
def sigmoid(z):
    """Numerically stable sigmoid."""
    return np.where(z >= 0,
                    1 / (1 + np.exp(-z)),
                    np.exp(z) / (1 + np.exp(z)))

def sigmoid_derivative(z):
    s = sigmoid(z)
    return s * (1 - s)

# ═══════════════════════════════════════
# 2. TANH
# ═══════════════════════════════════════
def tanh(z):
    return np.tanh(z)

def tanh_derivative(z):
    return 1 - np.tanh(z) ** 2

# ═══════════════════════════════════════
# 3. ReLU
# ═══════════════════════════════════════
def relu(z):
    return np.maximum(0, z)

def relu_derivative(z):
    return (z > 0).astype(np.float64)

# ═══════════════════════════════════════
# 4. LEAKY ReLU
# ═══════════════════════════════════════
def leaky_relu(z, alpha=0.01):
    return np.where(z > 0, z, alpha * z)

def leaky_relu_derivative(z, alpha=0.01):
    return np.where(z > 0, 1.0, alpha)

# ═══════════════════════════════════════
# 5. ELU
# ═══════════════════════════════════════
def elu(z, alpha=1.0):
    return np.where(z > 0, z, alpha * (np.exp(z) - 1))

def elu_derivative(z, alpha=1.0):
    return np.where(z > 0, 1.0, alpha * np.exp(z))

# ═══════════════════════════════════════
# 6. GELU (approximate)
# ═══════════════════════════════════════
def gelu(z):
    """Approximate GELU used in BERT/GPT."""
    return 0.5 * z * (1 + np.tanh(
        np.sqrt(2 / np.pi) * (z + 0.044715 * z**3)
    ))

def gelu_derivative(z):
    """Numerical derivative for simplicity."""
    h = 1e-7
    return (gelu(z + h) - gelu(z - h)) / (2 * h)

# ═══════════════════════════════════════
# 7. SWISH / SiLU
# ═══════════════════════════════════════
def swish(z):
    return z * sigmoid(z)

def swish_derivative(z):
    s = sigmoid(z)
    return s + z * s * (1 - s)

# ═══════════════════════════════════════
# 8. SOFTMAX
# ═══════════════════════════════════════
def softmax(z):
    """Numerically stable softmax."""
    z_shifted = z - np.max(z, axis=-1, keepdims=True)
    exp_z = np.exp(z_shifted)
    return exp_z / np.sum(exp_z, axis=-1, keepdims=True)

Visualize All Activations on One Plot

Python — Matplotlib
import matplotlib.pyplot as plt

z = np.linspace(-6, 6, 500)

activations = {
    'Sigmoid':    (sigmoid(z),    sigmoid_derivative(z),    '#6366f1'),
    'Tanh':       (tanh(z),       tanh_derivative(z),       '#0891b2'),
    'ReLU':       (relu(z),       relu_derivative(z),       '#16a34a'),
    'Leaky ReLU': (leaky_relu(z), leaky_relu_derivative(z), '#ea580c'),
    'ELU':        (elu(z),        elu_derivative(z),        '#0d9488'),
    'GELU':       (gelu(z),       gelu_derivative(z),       '#7c3aed'),
    'Swish':      (swish(z),      swish_derivative(z),      '#d946ef'),
}

fig, axes = plt.subplots(1, 2, figsize=(16, 6))

# Left: Activation functions
for name, (act, deriv, color) in activations.items():
    axes[0].plot(z, act, label=name, color=color, linewidth=2)
axes[0].set_title('Activation Functions', fontweight='bold')
axes[0].set_xlabel('z'); axes[0].set_ylabel('f(z)')
axes[0].axhline(y=0, color='gray', linestyle='--', alpha=0.5)
axes[0].axvline(x=0, color='gray', linestyle='--', alpha=0.5)
axes[0].legend(); axes[0].set_ylim(-2, 6)

# Right: Derivatives
for name, (act, deriv, color) in activations.items():
    axes[1].plot(z, deriv, label=f"{name}'(z)", color=color, linewidth=2)
axes[1].set_title('Derivatives (Gradient Flow)', fontweight='bold')
axes[1].set_xlabel('z'); axes[1].set_ylabel("f'(z)")
axes[1].axhline(y=0, color='gray', linestyle='--', alpha=0.5)
axes[1].axhline(y=1, color='gray', linestyle=':', alpha=0.4)
axes[1].legend(); axes[1].set_ylim(-0.5, 1.5)

plt.tight_layout()
plt.savefig('activation_functions_comparison.png', dpi=150)
plt.show()

Compare Gradient Flow Through 20 Layers

Python — Gradient Experiment
def gradient_flow_experiment(activation_fn, deriv_fn, n_layers=20, n_samples=1000):
    """Simulate gradient flow through n_layers with given activation."""
    np.random.seed(42)
    hidden_size = 64

    # He initialization for all layers
    gradients = []
    grad = np.ones(hidden_size)  # Start with gradient of 1.0

    for l in range(n_layers):
        # Random pre-activation values (simulating forward pass)
        z = np.random.randn(hidden_size) * np.sqrt(2.0 / hidden_size)
        # Multiply by local gradient (activation derivative)
        local_grad = deriv_fn(z)
        grad = grad * local_grad
        gradients.append(np.mean(np.abs(grad)))

    return gradients

# Run for each activation
results = {}
for name, deriv_fn in [('Sigmoid', sigmoid_derivative),
                       ('Tanh', tanh_derivative),
                       ('ReLU', relu_derivative),
                       ('Leaky ReLU', leaky_relu_derivative),
                       ('GELU', gelu_derivative),
                       ('Swish', swish_derivative)]:
    results[name] = gradient_flow_experiment(sigmoid if name == 'Sigmoid' else relu,
                                             deriv_fn)

# Plot gradient magnitude vs layer depth
plt.figure(figsize=(10, 6))
for name, grads in results.items():
    plt.plot(range(1, 21), grads, label=name, linewidth=2, marker='o', markersize=3)
plt.yscale('log')
plt.xlabel('Layer (from output to input)')
plt.ylabel('Mean |gradient|')
plt.title('Gradient Flow: Vanishing Gradient Demonstration')
plt.legend(); plt.grid(True, alpha=0.3)
plt.show()

Library Implementations — PyTorch & TensorFlow

PyTorch

PyTorch
import torch
import torch.nn as nn
import torch.nn.functional as F

z = torch.linspace(-6, 6, 100, requires_grad=True)

# All activations as one-liners
sig   = torch.sigmoid(z)               # or F.sigmoid(z)
tan   = torch.tanh(z)                  # or F.tanh(z)
rel   = F.relu(z)                      # or torch.relu(z)
lrel  = F.leaky_relu(z, 0.01)         # α = 0.01
elu_  = F.elu(z, alpha=1.0)            # α = 1.0
gel   = F.gelu(z)                      # exact or approximate='tanh'
swi   = F.silu(z)                      # SiLU = Swish(β=1)
sft   = F.softmax(z.unsqueeze(0), dim=-1)

# Using as nn.Module layers in a network
class FlexibleNet(nn.Module):
    def __init__(self, activation='relu'):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 10)

        # Activation selection
        act_map = {
            'relu':       nn.ReLU(),
            'leaky_relu': nn.LeakyReLU(0.01),
            'elu':        nn.ELU(alpha=1.0),
            'gelu':       nn.GELU(),
            'silu':       nn.SiLU(),        # Swish
            'sigmoid':    nn.Sigmoid(),
            'tanh':       nn.Tanh(),
            'prelu':      nn.PReLU(),       # learnable α
        }
        self.act = act_map[activation]

    def forward(self, x):
        x = self.act(self.fc1(x))
        x = self.act(self.fc2(x))
        return self.fc3(x)   # No activation on output (use CrossEntropyLoss)

# Compare activations on MNIST
for act_name in ['relu', 'sigmoid', 'gelu', 'silu']:
    model = FlexibleNet(activation=act_name)
    print(f"{act_name}: {sum(p.numel() for p in model.parameters())} params")

TensorFlow / Keras

TensorFlow / Keras
import tensorflow as tf
from tensorflow.keras import layers, models

# Build model with any activation
def build_model(activation='relu'):
    model = models.Sequential([
        layers.Dense(256, activation=activation, input_shape=(784,)),
        layers.Dense(128, activation=activation),
        layers.Dense(10, activation='softmax')
    ])
    return model

# Keras supports these strings directly:
# 'relu', 'sigmoid', 'tanh', 'elu', 'selu', 'gelu', 'swish'
# For LeakyReLU, use: layers.LeakyReLU(alpha=0.01)
# For PReLU: layers.PReLU()

# Custom activation example
@tf.function
def mish(z):
    """Mish activation: z * tanh(softplus(z))"""
    return z * tf.math.tanh(tf.math.softplus(z))

model = models.Sequential([
    layers.Dense(256, input_shape=(784,)),
    layers.Activation(mish),
    layers.Dense(10, activation='softmax')
])

Visual Diagrams

All Activations Side-by-Side

Gradient Flow Through a Deep Network

Softmax Visualization

Industry Case Studies

🇮🇳 India: Flipkart Product Categorization — ReLU vs Sigmoid in Hidden Layers

The One-Line Fix
# Before (bad)
self.hidden = nn.Sequential(
    nn.Linear(512, 256), nn.Sigmoid(),  # ❌ Sigmoid in hidden
    nn.Linear(256, 128), nn.Sigmoid(),
)

# After (good)
self.hidden = nn.Sequential(
    nn.Linear(512, 256), nn.BatchNorm1d(256), nn.ReLU(),  # ✅ ReLU + BN
    nn.Linear(256, 128), nn.BatchNorm1d(128), nn.ReLU(),
)

🇺🇸 Global: GPT Architecture — Why GELU Over ReLU in Transformers

Architecture
FFN(x) = W₂ · GELU(W₁ · x + b₁) + b₂

# In GPT-3 (175B parameters):
# W₁: [12288 × 49152]  (expand 4×)
# W₂: [49152 × 12288]  (project back)
# GELU applied to 49152-dimensional vector

PyTorch
class TransformerFFN(nn.Module):
    """Feed-forward network as used in GPT-2/3."""
    def __init__(self, d_model=768, d_ff=3072):
        super().__init__()
        self.fc1 = nn.Linear(d_model, d_ff)
        self.fc2 = nn.Linear(d_ff, d_model)
        self.gelu = nn.GELU()     # ← THE key choice
        self.dropout = nn.Dropout(0.1)

    def forward(self, x):
        x = self.gelu(self.fc1(x))  # Expand + activate
        x = self.dropout(self.fc2(x))  # Project back
        return x

Common Misconceptions

GATE / Exam Corner

Formula Sheet

GATE Previous Year Style Questions

Prediction Table — High-Probability GATE Topics

Interview Prep

Conceptual Questions

Coding Interview Question

Python — Interview Solution
def stable_softmax(z):
    """
    Numerically stable softmax.
    Args: z — numpy array of shape (batch_size, n_classes)
    Returns: probability distribution, same shape as z
    """
    # Subtract max for numerical stability (prevents overflow)
    z_shifted = z - np.max(z, axis=-1, keepdims=True)
    exp_z = np.exp(z_shifted)
    return exp_z / np.sum(exp_z, axis=-1, keepdims=True)

# Test
z = np.array([[1000, 1001, 1002]])  # Would overflow without stability trick
print(stable_softmax(z))
# Output: [[0.0900, 0.2447, 0.6652]]

# Verify: without stability
# np.exp(1000) = inf → NaN! ← Our version handles this.

Case Study Interview Question

Hands-On Lab / Mini-Project

🔬 Project: "The Great Activation Function Bake-Off"

Objective: Train the same neural network architecture with 7 different activation functions on the same dataset and compare: convergence speed, final accuracy, gradient health, and dead neuron count.

Setup

Python — Project Template
import torch
import torch.nn as nn
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# Dataset: Fashion-MNIST (10 classes, 28×28 grayscale)
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])
train_data = datasets.FashionMNIST('./data', train=True,
                                     download=True, transform=transform)
train_loader = DataLoader(train_data, batch_size=128, shuffle=True)

# Architecture: 5-layer MLP (same for all activations)
class ActivationTestNet(nn.Module):
    def __init__(self, act_fn):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(784, 512),  act_fn(),
            nn.Linear(512, 256),  act_fn(),
            nn.Linear(256, 128),  act_fn(),
            nn.Linear(128, 64),   act_fn(),
            nn.Linear(64, 10)
        )

    def forward(self, x):
        return self.layers(x.view(-1, 784))

# Activations to test
activations = {
    'Sigmoid':    nn.Sigmoid,
    'Tanh':       nn.Tanh,
    'ReLU':       nn.ReLU,
    'LeakyReLU': lambda: nn.LeakyReLU(0.01),
    'ELU':        nn.ELU,
    'GELU':       nn.GELU,
    'SiLU':       nn.SiLU,  # Swish
}

# Training loop (per activation)
def train_and_evaluate(act_name, act_fn, epochs=20):
    model = ActivationTestNet(act_fn)
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
    criterion = nn.CrossEntropyLoss()
    history = {'loss': [], 'acc': [], 'grad_norms': []}

    for epoch in range(epochs):
        total_loss, correct, total = 0, 0, 0
        for X, y in train_loader:
            out = model(X)
            loss = criterion(out, y)
            optimizer.zero_grad()
            loss.backward()

            # Record gradient norms
            grad_norm = sum(p.grad.norm().item()
                          for p in model.parameters()
                          if p.grad is not None)
            optimizer.step()
            total_loss += loss.item()
            correct += (out.argmax(1) == y).sum().item()
            total += y.size(0)

        history['loss'].append(total_loss / len(train_loader))
        history['acc'].append(correct / total)
        history['grad_norms'].append(grad_norm)
        print(f"{act_name} Epoch {epoch+1}: Loss={history['loss'][-1]:.4f}, Acc={history['acc'][-1]:.4f}")
    return history

# Run all experiments
all_results = {}
for name, fn in activations.items():
    print(f"\n{'='*50}\nTraining with {name}\n{'='*50}")
    all_results[name] = train_and_evaluate(name, fn)

Rubric (100 points)

Exercises (22 Questions)

Section A: Conceptual (5 Questions)

Section B: Mathematical (8 Questions)

Section C: Coding (4 Questions)

Section D: Critical Thinking (3 Questions)

★ Starred Research Questions (2 Questions)

Connections

Chapter Summary

Further Reading

🇮🇳 Indian Resources

• NPTEL: Deep Learning (IIT Madras, Prof. Mitesh Khapra) — Weeks 3-4 cover activation functions with excellent Hindi/English explanations. Free on Swayam.
• GATE CS/DA Previous Year Papers: 2022-2025 papers include activation function questions. Download from gate.iitk.ac.in.
• "Deep Learning" by S. Haykin (Pearson India): Chapter 4 covers activation functions with mathematical rigor suitable for GATE preparation.
• NPTEL: Machine Learning (IIT Kharagpur, Prof. Sudeshna Sarkar) — Lecture on neural networks covers sigmoid and tanh in detail.

🌍 Global Resources

• Paper: "Rectified Linear Units Improve Restricted Boltzmann Machines" — Nair & Hinton, ICML 2010. The ReLU origin story.
• Paper: "Gaussian Error Linear Units (GELUs)" — Hendrycks & Gimpel, 2016. The activation behind Transformers.
• Paper: "Searching for Activation Functions" — Ramachandran, Zoph, Le (Google Brain, 2017). How Swish was discovered.
• Paper: "Delving Deep into Rectifiers" (PReLU) — He et al., ICCV 2015. He initialization + PReLU.
• 3Blue1Brown — "But what is a neural network?" (YouTube): Beautiful visualization of how activation functions transform linear spaces into non-linear ones.
• Distill.pub: Multiple articles on feature visualization that show what different activations learn.
• CS231n (Stanford) Notes: "Neural Networks Part 1" has an excellent section on activation functions with pros/cons.

Interactive Tools

• TensorFlow Playground: playground.tensorflow.org — Toggle between activations and watch how decision boundaries change
• Desmos Graphing Calculator: desmos.com — Plot any activation function interactively to build intuition
• PyTorch Documentation: pytorch.org/docs — Complete list of all supported activations with mathematical definitions