Phase 4 • EduArtha

Deep Learning

The engine behind modern AI. Neural networks, backpropagation, and deep architectures are built here. Every breakthrough from GPT to Stable Diffusion to AlphaFold relies on these foundations.

⏱ 4–8 months  |  14 Chapters  |  50+ Exercises

Part I

Neural Network Fundamentals

The building blocks of every deep learning model

Chapter 1

Perceptrons & Multilayer Networks

Learning Objectives

  • Understand the perceptron — the simplest neural unit
  • Build multilayer perceptrons (MLPs) from scratch
  • Grasp universal approximation — why depth matters
  • Connect neurons to modern AI: every LLM is built on these

The Perceptron

A perceptron computes a weighted sum of inputs, adds a bias, and passes through an activation function. It's a single artificial neuron — the atom of deep learning.

output = activation(w₁x₁ + w₂x₂ + ... + wₙxₙ + b) = activation(W·X + b)
Python
import numpy as np

class Perceptron:
    def __init__(self, n_inputs, lr=0.01):
        self.weights = np.random.randn(n_inputs) * 0.01
        self.bias = 0.0
        self.lr = lr

    def forward(self, x):
        return 1.0 if np.dot(self.weights, x) + self.bias > 0 else 0.0

    def train(self, X, y, epochs=100):
        for _ in range(epochs):
            for xi, yi in zip(X, y):
                pred = self.forward(xi)
                error = yi - pred
                self.weights += self.lr * error * xi
                self.bias += self.lr * error

# AND gate — linearly separable
X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([0, 0, 0, 1])
p = Perceptron(2)
p.train(X, y)
print([p.forward(xi) for xi in X])  # [0, 0, 0, 1] ✓

Multilayer Perceptron (MLP)

Stacking layers of neurons creates an MLP — capable of learning any continuous function (Universal Approximation Theorem). The key insight: non-linear activations between layers allow the network to model complex, non-linear relationships.

Python
import torch
import torch.nn as nn

class MLP(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, output_dim)
        )

    def forward(self, x):
        return self.net(x)

model = MLP(784, 256, 10)  # MNIST: 784 pixels → 10 digits
print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

Why This Matters for AI

Every modern AI model — GPT-4, Gemini, DALL-E, AlphaFold — is built from layers of neurons. The MLP is the fundamental building block. Transformer feed-forward layers? MLPs. Classification heads? MLPs. Understanding how neurons combine to learn representations is the foundation for everything that follows.

Project: MNIST Digit Classifier from Scratch

Python
import torch, torch.nn as nn
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# Data
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,),(0.3081,))])
train_data = datasets.MNIST('./data', train=True, download=True, transform=transform)
test_data = datasets.MNIST('./data', train=False, transform=transform)
train_loader = DataLoader(train_data, batch_size=64, shuffle=True)
test_loader = DataLoader(test_data, batch_size=1000)

# Model
model = nn.Sequential(
    nn.Flatten(),
    nn.Linear(784, 512), nn.ReLU(), nn.Dropout(0.2),
    nn.Linear(512, 256), nn.ReLU(), nn.Dropout(0.2),
    nn.Linear(256, 10)
)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

# Train
for epoch in range(5):
    model.train()
    for X, y in train_loader:
        loss = criterion(model(X), y)
        optimizer.zero_grad(); loss.backward(); optimizer.step()

    # Evaluate
    model.eval()
    correct = sum(
        (model(X).argmax(1) == y).sum().item()
        for X, y in test_loader)
    print(f"Epoch {epoch+1}: Accuracy = {correct/len(test_data):.2%}")
# Achieves ~98% accuracy!

Exercises

Exercise 1.1: Why can't a single perceptron solve XOR?

XOR is not linearly separable — no single line can divide the four points into correct classes. A perceptron draws one linear boundary. You need at least 2 layers (hidden + output) to create the two boundaries needed for XOR. This limitation motivated the development of MLPs.

Exercise 1.2: How many parameters does a network with layers [784, 512, 256, 10] have?

Layer 1: 784×512 + 512 = 401,920. Layer 2: 512×256 + 256 = 131,328. Layer 3: 256×10 + 10 = 2,570. Total: 535,818. Each layer has weights (input×output) plus biases (output). Modern LLMs have billions — but the math is the same.

Exercise 1.3: What is the Universal Approximation Theorem?

A feedforward network with a single hidden layer of sufficient width can approximate any continuous function to any desired accuracy. However, deep networks (many layers) achieve this with far fewer parameters than wide-shallow ones. Depth enables hierarchical feature learning — edges → shapes → objects.

Chapter Summary

  • Perceptrons compute weighted sums with activation — the basic neural unit
  • MLPs stack layers with non-linear activations to learn complex functions
  • Every modern AI model is built from these fundamental building blocks
  • Depth enables hierarchical feature learning — the key insight of deep learning