Mathematical Toolkit for Deep Learning

Every neural network is a composition of mathematical functions. Before you write a single line of PyTorch, you must own the math that powers it — linear algebra for data flow, calculus for learning, probability for uncertainty, and information theory for loss functions.

Learning Objectives

The Hook: Why Math Matters

2.1 Linear Algebra for Deep Learning

Linear algebra is the language of data in deep learning. Every input, every weight, every output is a multi-dimensional array. Let's build the vocabulary.

2.1.1 Scalars, Vectors, Matrices, and Tensors

Scalar (0-D Tensor)

A single number. We denote scalars with lowercase italic letters: x, y, α, λ.

Vector (1-D Tensor)

An ordered list of numbers. We denote vectors with bold lowercase: x. Each element is x₁, x₂, ..., xₙ.

Matrix (2-D Tensor)

A 2-D array of numbers. Denoted with bold uppercase: A. Shape is m × n (m rows, n columns).

Tensor (n-D Array)

An array with more than 2 axes. A color image is a 3-D tensor (height × width × channels). A batch of images is a 4-D tensor.

2.1.2 Matrix Multiplication: The Dot Product

Matrix multiplication is the single most important operation in neural networks. Every forward pass is a series of matrix multiplications.

Intuition: Each element C_{ij} is the dot product of row i of A with column j of B. Think of it as: "How much does row i of A align with column j of B?"

In neural networks: If x is a single input vector (1×n) and W is a weight matrix (n×h), then x·W gives the output (1×h) — one number per hidden neuron. Each output neuron computes a weighted sum of the inputs.

2.1.3 Transpose

2.1.4 Matrix Inverse

2.1.5 Element-wise (Hadamard) Product

Don't confuse this with matrix multiplication! The Hadamard product multiplies corresponding elements, while matrix multiplication computes dot products of rows and columns.

Data
Passenger Matrix X ∈ ℝ⁵ˣ⁴:

         Age    Fare(₹)  Distance(km)  Class
P1  [    28,    1450,      850,          3   ]
P2  [    45,    3200,     1200,          2   ]
P3  [    32,    5800,     2100,          1   ]
P4  [    56,    1800,      950,          3   ]
P5  [    23,    4500,     1800,          1   ]

Shape: (5, 4)
X[2, 1] = 5800   (3rd passenger, 2nd feature = Fare)
X[:, 0] = [28, 45, 32, 56, 23]  (all ages — a column vector)

2.2 Calculus for Deep Learning

Calculus tells us how to learn. A neural network improves by computing how much each weight contributed to the error, then adjusting it. That computation is a derivative.

2.2.1 Derivatives from First Principles

Example: Deriving d/dx [x²] from first principles

So d/dx [x²] = 2x. At x = 3, the slope is 6 — meaning a tiny increase in x increases x² by approximately 6 times that increase.

Key Derivative Rules

2.2.2 The Chain Rule — Heart of Backpropagation

Why this matters: A neural network is a composition of functions: output = f₃(f₂(f₁(x))). To compute how the loss changes w.r.t. weight w₁ in the first layer, we chain derivatives through every subsequent layer. This is backpropagation.

Example: Chain Rule

Let y = (3x + 2)⁵. Find dy/dx.

2.2.3 Partial Derivatives and Gradients

When a function has multiple inputs (like a loss function with many weights), we take partial derivatives — differentiate w.r.t. one variable while holding others constant.

The Gradient Vector

2.2.4 Jacobian and Hessian (Brief Introduction)

2.2.5 Worked Example: Derivative of the Sigmoid Function

2.3 Probability & Statistics for Deep Learning

Neural networks don't output certainties — they output probabilities. Understanding probability is essential for designing loss functions, interpreting outputs, and reasoning about uncertainty.

2.3.1 Bernoulli Distribution

Deep learning connection: Binary classification IS a Bernoulli distribution. When your model outputs P(churn) = 0.82 for a Paytm user, it's saying: "This user's churn follows Bernoulli(0.82)."

2.3.2 Gaussian (Normal) Distribution

Deep learning connection: Weight initialization (Xavier, He) draws from Gaussians. Noise in VAEs is Gaussian. Regression targets are often modeled as Gaussian with learned mean and variance.

2.3.3 Conditional Probability & Bayes' Theorem

2.3.4 Maximum Likelihood Estimation (MLE)

MLE answers: "Given the data we observed, what parameter values make the data most probable?"

2.3.5 MLE for Bernoulli: Full Derivation

2.3.6 From MLE to Cross-Entropy Loss

Here's the most important derivation in this chapter — why we use cross-entropy as a loss function.

Cross-entropy is not an arbitrary choice — it emerges naturally from maximum likelihood estimation under a Bernoulli model. When someone says "we use cross-entropy loss for classification," they're saying "we're doing MLE."

2.4 Information Theory for Deep Learning

Information theory, pioneered by Claude Shannon (1948), gives us a mathematical framework for quantifying uncertainty and information. It provides the theoretical foundation for why cross-entropy is the natural loss function for classification.

2.4.1 Entropy: Measuring Uncertainty

Intuition: Entropy measures how "surprised" you are on average. A fair coin (p=0.5) has maximum entropy — you're maximally uncertain. A biased coin (p=0.99) has low entropy — you know it'll be heads.

Examples

2.4.2 KL Divergence: Measuring Distribution Difference

Intuition: KL divergence measures how much "extra information" you need when you use distribution q to approximate distribution p. It's the "penalty" for using the wrong distribution.

2.4.3 Cross-Entropy: The Natural Loss

Numerical Example: Why Cross-Entropy > MSE for Classification

True label: y = 1. Model predicts ŷ = 0.01 (confidently wrong!).

Cross-entropy gives a gradient of −100 vs MSE's −1.98. When the model is confidently wrong, cross-entropy punishes it 50× harder and pushes stronger corrections. That's why training converges faster with cross-entropy.

NumPy Code Lab: Math from Scratch

Let's implement every key operation in NumPy. This is where formulas become executable code.

4.1 Linear Algebra Operations

Python
import numpy as np

# ────────────────── SCALARS, VECTORS, MATRICES, TENSORS ──────────────────

# Scalar
learning_rate = 0.01

# Vector — 5 features of a Paytm user
user = np.array([25, 12500, 3, 0.85, 450])
print(f"User vector shape: {user.shape}")   # (5,)

# Matrix — 100 users × 5 features
np.random.seed(42)
X = np.random.randn(100, 5)
print(f"Data matrix shape: {X.shape}")    # (100, 5)

# Tensor — batch of 32 color images (28×28)
images = np.random.randn(32, 28, 28, 3)
print(f"Image batch shape: {images.shape}")  # (32, 28, 28, 3)

# ────────────────── DOT PRODUCT ──────────────────

# Vector dot product: weighted sum of features
weights = np.array([0.3, 0.0001, -0.5, 2.0, 0.002])
score = np.dot(user, weights)
print(f"Churn score: {score:.4f}")

# Matrix multiplication: all users at once
W = np.random.randn(5, 3)    # 5 features → 3 hidden neurons
H = X @ W                      # (100,5) @ (5,3) = (100,3)
print(f"Hidden layer shape: {H.shape}")  # (100, 3)

# ────────────────── TRANSPOSE ──────────────────
A = np.array([[1, 2, 3],
              [4, 5, 6]])
print(f"A shape: {A.shape}")          # (2, 3)
print(f"A^T shape: {A.T.shape}")      # (3, 2)

# Verify (AB)^T = B^T A^T
B = np.random.randn(3, 4)
lhs = (A @ B).T
rhs = B.T @ A.T
print(f"(AB)^T == B^T A^T: {np.allclose(lhs, rhs)}")  # True

# ────────────────── INVERSE ──────────────────
M = np.array([[4, 7], [2, 6]])
M_inv = np.linalg.inv(M)
print(f"M · M⁻¹ =\n{M @ M_inv}")     # Identity matrix

# ────────────────── HADAMARD (ELEMENT-WISE) PRODUCT ──────────────────
gate = np.array([[1, 0, 1],
                 [0, 1, 0]])   # Binary mask (like dropout)
data = np.array([[5, 3, 8],
                 [2, 7, 4]])
masked = gate * data            # Hadamard product: * in NumPy
print(f"Hadamard:\n{masked}")     # [[5, 0, 8], [0, 7, 0]]

4.2 Calculus: Numerical Gradient & Sigmoid

Python
import numpy as np

# ────────────────── SIGMOID AND ITS DERIVATIVE ──────────────────

def sigmoid(z):
    """σ(z) = 1 / (1 + e^(-z))"""
    return 1 / (1 + np.exp(-z))

def sigmoid_derivative(z):
    """σ'(z) = σ(z)(1 - σ(z))"""
    s = sigmoid(z)
    return s * (1 - s)

# Verify: maximum of sigmoid derivative is at z=0
z_values = np.linspace(-6, 6, 1000)
derivs = sigmoid_derivative(z_values)
print(f"Max σ'(z) = {derivs.max():.4f} at z = {z_values[derivs.argmax()]:.2f}")
# Max σ'(z) = 0.2500 at z = 0.00

# ────────────────── NUMERICAL GRADIENT ──────────────────

def numerical_gradient(f, x, h=1e-7):
    """Compute gradient numerically (central difference)"""
    grad = np.zeros_like(x)
    for i in range(len(x)):
        x_plus = x.copy(); x_plus[i] += h
        x_minus = x.copy(); x_minus[i] -= h
        grad[i] = (f(x_plus) - f(x_minus)) / (2 * h)
    return grad

# Test: f(x,y) = x² + 3xy → ∂f/∂x = 2x+3y, ∂f/∂y = 3x
def test_func(v):
    x, y = v[0], v[1]
    return x**2 + 3*x*y

point = np.array([2.0, 5.0])
num_grad = numerical_gradient(test_func, point)
analytical_grad = np.array([2*2 + 3*5, 3*2])  # [19, 6]

print(f"Numerical gradient:  {num_grad}")
print(f"Analytical gradient: {analytical_grad}")
print(f"Match: {np.allclose(num_grad, analytical_grad)}")  # True

# ────────────────── GRADIENT DESCENT DEMO ──────────────────

def loss(w):
    """Simple quadratic loss: L = (w - 3)²"""
    return (w - 3)**2

w = 0.0           # Start at w=0
lr = 0.1          # Learning rate
print(f"{'Step':>4} {'w':>8} {'Loss':>10}")
for step in range(20):
    grad = 2 * (w - 3)   # dL/dw = 2(w-3)
    w = w - lr * grad      # Gradient descent update
    print(f"{step+1:>4} {w:>8.4f} {loss(w):>10.6f}")
# w converges to 3.0 (the minimum)

4.3 Probability: Distributions, MLE, Cross-Entropy

Python
import numpy as np

# ────────────────── BERNOULLI DISTRIBUTION ──────────────────

def bernoulli_pmf(x, p):
    """P(X=x) = p^x * (1-p)^(1-x)"""
    return (p ** x) * ((1 - p) ** (1 - x))

# PhonePe fraud example: p = 0.047
p_fraud = 0.047
print(f"P(fraud)     = {bernoulli_pmf(1, p_fraud):.4f}")   # 0.047
print(f"P(not fraud) = {bernoulli_pmf(0, p_fraud):.4f}")   # 0.953

# ────────────────── GAUSSIAN DISTRIBUTION ──────────────────

def gaussian_pdf(x, mu, sigma):
    """p(x) = (1/√(2πσ²)) exp(-(x-μ)²/(2σ²))"""
    coeff = 1 / np.sqrt(2 * np.pi * sigma**2)
    exponent = -((x - mu)**2) / (2 * sigma**2)
    return coeff * np.exp(exponent)

# Zomato delivery time: mean 35 min, std 8 min
x = np.array([25, 30, 35, 40, 50])
probs = gaussian_pdf(x, mu=35, sigma=8)
for xi, pi in zip(x, probs):
    print(f"P(delivery={xi} min) = {pi:.4f}")

# ────────────────── MLE FOR BERNOULLI ──────────────────

# Simulate 1000 transactions, 47 fraudulent
np.random.seed(42)
transactions = np.zeros(1000)
transactions[:47] = 1
np.random.shuffle(transactions)

# MLE estimate: p_hat = sum(x) / n
p_hat = transactions.mean()
print(f"\nMLE estimate of fraud probability: {p_hat:.4f}")  # 0.047

# Log-likelihood at p_hat
log_lik = np.sum(transactions * np.log(p_hat) +
                 (1 - transactions) * np.log(1 - p_hat))
print(f"Log-likelihood at p_hat: {log_lik:.2f}")

# ────────────────── CROSS-ENTROPY LOSS ──────────────────

def binary_cross_entropy(y_true, y_pred, eps=1e-15):
    """BCE = -(1/n) Σ [y log(ŷ) + (1-y) log(1-ŷ)]"""
    y_pred = np.clip(y_pred, eps, 1 - eps)  # Numerical stability
    return -np.mean(
        y_true * np.log(y_pred) +
        (1 - y_true) * np.log(1 - y_pred)
    )

# Test: true labels vs model predictions
y_true = np.array([1, 0, 1, 1, 0])
y_good = np.array([0.9, 0.1, 0.85, 0.95, 0.05])  # Good model
y_bad  = np.array([0.3, 0.8, 0.4, 0.2, 0.7])   # Bad model

print(f"\nGood model BCE: {binary_cross_entropy(y_true, y_good):.4f}")
print(f"Bad model  BCE: {binary_cross_entropy(y_true, y_bad):.4f}")
# Good model has LOWER loss → cross-entropy works!

# ────────────────── ENTROPY & KL DIVERGENCE ──────────────────

def entropy(p):
    """H(p) = -Σ p(x) log p(x)"""
    p = p[p > 0]  # Avoid log(0)
    return -np.sum(p * np.log(p))

def kl_divergence(p, q, eps=1e-15):
    """D_KL(p || q) = Σ p(x) log(p(x)/q(x))"""
    q = np.clip(q, eps, 1.0)
    return np.sum(p * np.log(p / q))

def cross_entropy(p, q, eps=1e-15):
    """H(p, q) = -Σ p(x) log q(x)"""
    q = np.clip(q, eps, 1.0)
    return -np.sum(p * np.log(q))

# True distribution vs two models
p_true = np.array([0.7, 0.2, 0.1])   # 3-class problem
q_good = np.array([0.65, 0.25, 0.1])
q_bad  = np.array([0.2, 0.3, 0.5])

print(f"\nEntropy H(p):        {entropy(p_true):.4f}")
print(f"KL(p||q_good):       {kl_divergence(p_true, q_good):.4f}")
print(f"KL(p||q_bad):        {kl_divergence(p_true, q_bad):.4f}")
print(f"CrossEnt(p, q_good): {cross_entropy(p_true, q_good):.4f}")
print(f"CrossEnt(p, q_bad):  {cross_entropy(p_true, q_bad):.4f}")
print(f"Verify: H(p) + KL(p||q_good) = {entropy(p_true) + kl_divergence(p_true, q_good):.4f}")
# Should equal cross_entropy(p_true, q_good)

Visual Walkthrough: Matrix Multiplication

Let's trace through a 3×3 matrix multiplication by hand, step by step.

Worked Examples: Hand Calculations

Calculation
X = ┌             ┐       W = ┌      ┐
    │ 12   399  45│           │ 0.1  │
    │  5   199  80│           │ 0.005│    (churn weight vector)
    │ 25   599  20│           │-0.02 │
    └             ┘           └      ┘

Common Mistakes & Pitfalls

Exercises

Section A: Multiple Choice Questions

Section B: Hand Calculation Problems

Section D: Programming Problems

Chapter Summary

Cheat Sheet: Formulas You Must Remember

References & Further Reading

Primary Textbooks

1. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning, Chapter 2 (Linear Algebra), Chapter 3 (Probability), Chapter 4 (Numerical Computation). MIT Press. deeplearningbook.org
2. Bishop, C. M. (2006). Pattern Recognition and Machine Learning, Chapter 1 (Introduction: Probability). Springer.
3. Strang, G. (2019). Linear Algebra and Learning from Data. Wellesley-Cambridge Press.

Online Resources

1. 3Blue1Brown — Essence of Linear Algebra (YouTube Series). Exceptional visual intuition for vectors, transformations, and eigenvalues.
2. 3Blue1Brown — Essence of Calculus (YouTube Series). Derivatives and integrals explained visually.
3. Khan Academy — Multivariable Calculus. Gradients, partial derivatives, Jacobians.
4. Stanford CS229 Notes — Linear Algebra Review. Concise reference for ML-relevant linear algebra.
5. colah's blog — Visual Information Theory (2015). Beautiful explanation of entropy, cross-entropy, and KL divergence. colah.github.io

Indian Context

1. NPTEL — Mathematics for Machine Learning by IIT Madras. Free video lectures covering all topics in this chapter.
2. NPTEL — Deep Learning by Prof. Mitesh Khapra, IIT Madras. Mathematical foundations in Weeks 1-3.
3. UPI Transaction Statistics — NPCI. npci.org.in