Chapter 3: Logistic Regression as a Neural Network

Bloom's Taxonomy Progression

Learning Objectives

By the end of this chapter, you will be able to:

• Explain why the step-function perceptron needs to be replaced with a smooth, differentiable activation for learning via gradient descent
• Derive the sigmoid function σ(z) = 1/(1+e^−z) and prove all five key properties: range, midpoint, symmetry, limits, and derivative σ′ = σ(1−σ)
• Recognize logistic regression as a single-neuron neural network and draw its computational architecture
• Derive Binary Cross-Entropy from scratch: Bernoulli → Likelihood → Log-Likelihood → Negate → BCE
• Compute the cost function J(w,b) = average BCE over m training examples
• Derive gradient descent update rules ∂J/∂w and ∂J/∂b using the chain rule on the computation graph
• Implement a complete LogisticRegression class from scratch with sigmoid, loss, forward, backward, update, fit, predict, and accuracy methods
• Train the model on a synthetic Indian bank loan dataset, plot the loss curve and decision boundary
• Compare vectorized (NumPy) vs loop-based implementations with timing benchmarks
• Execute a full forward + backward pass by hand on a worked example with 2 features and 3 samples

Opening Hook — The 12-Second Loan Decision

The Intuition First — From Step to Smooth

The Dimmer Switch Analogy

In Chapter 2 (your math toolkit), you saw that a perceptron uses a step function: output 1 if the weighted sum exceeds a threshold, output 0 otherwise. Think of it as a light switch — it's either ON or OFF, nothing in between.

But here's the problem: you can't learn from a switch. Imagine you're adjusting the temperature dial on your geyser. If the dial only had two positions — "freezing" and "boiling" — you'd never find a comfortable temperature. You need a smooth dial that lets you make tiny adjustments and see small changes in temperature. That smooth dial is the sigmoid function.

The "Aha!" Question

Ask yourself: Why can't gradient descent work with a step function? Because the derivative of a step function is zero everywhere (except at the discontinuity where it's undefined). If the derivative is zero, the gradient is zero, and the weight update Δw = −α·∂L/∂w = −α·0 = 0. The model never learns! You need a function with a non-zero derivative at every point. Enter the sigmoid.

Mathematical Foundation — Derived from First Principles

We build the entire logistic regression framework in five rigorous steps. Every equation is derived, not memorized. If you're confused at any step, you're thinking correctly — stay with it.

4a. The Sigmoid Function σ(z) — From Linear to Probability

The Problem: Unbounded Outputs

A neuron computes z = w^Tx + b, where z ∈ (−∞, +∞). But for binary classification, you need a probability — a number in [0, 1]. You need a function f: ℝ → (0, 1) that is smooth, monotonically increasing, and differentiable.

Definition

Sigmoid Value Table — Memorize These!

Logistic Regression = Single-Neuron Neural Network

Now here's the key insight. A logistic regression model is exactly a neural network with:

• One input layer with n_x features
• One output neuron with sigmoid activation
• No hidden layers

4b. Binary Cross-Entropy — Derived from Maximum Likelihood

We have a model that outputs ŷ = σ(w^Tx + b) ∈ (0, 1). We need a loss function — a way to quantify "how wrong" the model is. We don't pull this loss out of thin air. We derive it from the principle of Maximum Likelihood Estimation (MLE).

Why BCE Works: Intuitive Analysis

4c. The Cost Function J(w, b)

For a single training example (x, y), the loss is:

The cost function is the average loss over all m training examples:

Where ŷ⁽ⁱ⁾ = σ(w^Tx⁽ⁱ⁾ + b) for each sample i. The goal of training: find w* and b* that minimize J(w, b).

4d. Gradient Computation — ∂J/∂w and ∂J/∂b via Chain Rule

To minimize J, we need its gradients with respect to w and b. We'll derive these using the chain rule, one link at a time.

Gradient Descent Update Rules

With the gradients computed, we update parameters in the direction that decreases J:

Repeat for T iterations (epochs). Each iteration, the loss decreases (if α is small enough), and the model gets better at predicting.

4e. The Computation Graph — Forward and Backward Pass

A computation graph makes the chain rule visual. Each node is an operation. Forward pass flows left→right, computing outputs. Backward pass flows right→left, computing gradients.

Worked Examples — By Hand, Then by India, Then by World

Example 1: Full Forward + Backward Pass By Hand (2 features, 3 samples)

Example 2: 🇮🇳 SBI CIBIL Score Loan Approval

Example 3: 🌍 Google Ads Click-Through Rate (CTR) Prediction

Python Implementation — From Scratch + Library

6a. From-Scratch NumPy Implementation

Python
import numpy as np
import matplotlib.pyplot as plt
import time

class LogisticRegression:
    """
    Logistic Regression from scratch — a single-neuron neural network.
    Implements: sigmoid, BCE loss, forward, backward, update, fit, predict.
    """

    def __init__(self, n_features, learning_rate=0.01):
        # Initialize weights to small random values, bias to zero
        self.w = np.zeros((n_features, 1))   # shape (n, 1)
        self.b = 0.0
        self.lr = learning_rate
        self.losses = []

    def sigmoid(self, z):
        """Numerically stable sigmoid."""
        return np.where(
            z >= 0,
            1 / (1 + np.exp(-z)),
            np.exp(z) / (1 + np.exp(z))
        )

    def forward(self, X):
        """
        Forward pass: X → z → ŷ
        X: shape (n, m) — each column is a sample
        Returns: ŷ of shape (1, m)
        """
        self.z = np.dot(self.w.T, X) + self.b   # (1, m)
        self.y_hat = self.sigmoid(self.z)         # (1, m)
        return self.y_hat

    def compute_loss(self, Y):
        """
        Binary Cross-Entropy: J = -(1/m) Σ [y·log(ŷ) + (1-y)·log(1-ŷ)]
        Y: shape (1, m)
        """
        m = Y.shape[1]
        epsilon = 1e-8  # prevent log(0)
        cost = -(1/m) * np.sum(
            Y * np.log(self.y_hat + epsilon) +
            (1 - Y) * np.log(1 - self.y_hat + epsilon)
        )
        return np.squeeze(cost)

    def backward(self, X, Y):
        """
        Backward pass: compute gradients ∂J/∂w and ∂J/∂b
        The beautiful result: dz = ŷ - y
        """
        m = Y.shape[1]
        dz = self.y_hat - Y                        # (1, m)
        self.dw = (1/m) * np.dot(X, dz.T)           # (n, 1)
        self.db = (1/m) * np.sum(dz)                # scalar

    def update(self):
        """Gradient descent step: w := w - α·dw, b := b - α·db"""
        self.w -= self.lr * self.dw
        self.b -= self.lr * self.db

    def fit(self, X, Y, epochs=1000, print_every=100):
        """
        Full training loop.
        X: (n, m), Y: (1, m)
        """
        for i in range(epochs):
            # Forward
            self.forward(X)
            # Loss
            loss = self.compute_loss(Y)
            self.losses.append(loss)
            # Backward
            self.backward(X, Y)
            # Update
            self.update()
            # Print
            if i % print_every == 0:
                print(f"Epoch {i:4d} | Loss: {loss:.6f}")

    def predict(self, X, threshold=0.5):
        """Return binary predictions."""
        y_hat = self.forward(X)
        return (y_hat >= threshold).astype(int)

    def accuracy(self, X, Y):
        """Compute classification accuracy."""
        preds = self.predict(X)
        return np.mean(preds == Y) * 100

6b. Synthetic Indian Bank Loan Dataset + Training

Python
# ── Generate Synthetic SBI Loan Dataset ──
np.random.seed(42)
m = 500  # 500 loan applications

# Feature 1: CIBIL score (normalized to 0-1)
cibil = np.random.beta(5, 2, m)  # skewed toward higher scores
# Feature 2: EMI-to-income ratio (0-0.8)
emi_ratio = np.random.beta(2, 5, m) * 0.8

# Labels: P(default) is LOW when CIBIL is HIGH and EMI is LOW
z_true = -4.0 * cibil + 5.0 * emi_ratio + 0.5
prob_default = 1 / (1 + np.exp(-z_true))
y = (np.random.rand(m) < prob_default).astype(float)

# Stack into (n, m) format
X = np.vstack([cibil, emi_ratio])   # shape (2, 500)
Y = y.reshape(1, -1)                # shape (1, 500)

print(f"Dataset: {X.shape[1]} samples, {X.shape[0]} features")
print(f"Default rate: {Y.mean()*100:.1f}%")

# ── Train the model ──
model = LogisticRegression(n_features=2, learning_rate=1.0)
model.fit(X, Y, epochs=2000, print_every=200)

print(f"\nFinal Loss: {model.losses[-1]:.6f}")
print(f"Training Accuracy: {model.accuracy(X, Y):.2f}%")
print(f"Learned weights: w1={model.w[0,0]:.4f}, w2={model.w[1,0]:.4f}, b={model.b:.4f}")

6c. Plot Loss Curve + Decision Boundary

Python
# ── Plot 1: Loss Curve ──
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))

ax1.plot(model.losses, color='#7c3aed', linewidth=2)
ax1.set_xlabel('Epoch', fontsize=12)
ax1.set_ylabel('BCE Loss', fontsize=12)
ax1.set_title('Training Loss Curve', fontsize=14, fontweight='bold')
ax1.grid(True, alpha=0.3)

# ── Plot 2: Decision Boundary ──
# Create mesh grid
x1_range = np.linspace(0, 1, 200)
x2_range = np.linspace(0, 0.8, 200)
X1, X2 = np.meshgrid(x1_range, x2_range)
X_mesh = np.vstack([X1.ravel(), X2.ravel()])

# Predict on mesh
Z_mesh = model.forward(X_mesh).reshape(X1.shape)

# Plot
ax2.contourf(X1, X2, Z_mesh, levels=50, cmap='RdYlGn_r', alpha=0.8)
ax2.contour(X1, X2, Z_mesh, levels=[0.5], colors='black', linewidths=2)
scatter = ax2.scatter(X[0], X[1], c=Y.ravel(), cmap='RdYlGn_r',
                       edgecolors='black', s=20, alpha=0.6)
ax2.set_xlabel('CIBIL Score (normalized)', fontsize=12)
ax2.set_ylabel('EMI-to-Income Ratio', fontsize=12)
ax2.set_title('Decision Boundary — SBI Loan Default', fontsize=14, fontweight='bold')
plt.colorbar(scatter, ax=ax2, label='P(Default)')

plt.tight_layout()
plt.savefig('ch03_logistic_regression_plots.png', dpi=150)
plt.show()

6d. Vectorized vs Non-Vectorized — Timing Comparison

Python
# ── Non-vectorized (loop-based) gradient computation ──
def compute_gradients_loop(X, Y, w, b):
    """Compute gradients using explicit Python loops — SLOW!"""
    n, m = X.shape
    dw = np.zeros((n, 1))
    db = 0.0

    for i in range(m):
        x_i = X[:, i].reshape(-1, 1)    # (n, 1)
        z_i = np.dot(w.T, x_i) + b       # scalar
        y_hat_i = 1 / (1 + np.exp(-z_i)) # scalar
        dz_i = y_hat_i - Y[0, i]         # scalar
        dw += x_i * dz_i
        db += dz_i

    dw /= m
    db /= m
    return dw, db

# ── Vectorized gradient computation ──
def compute_gradients_vectorized(X, Y, w, b):
    """Compute gradients using NumPy broadcasting — FAST!"""
    m = X.shape[1]
    z = np.dot(w.T, X) + b                # (1, m)
    y_hat = 1 / (1 + np.exp(-z))          # (1, m)
    dz = y_hat - Y                         # (1, m)
    dw = (1/m) * np.dot(X, dz.T)            # (n, 1)
    db = (1/m) * np.sum(dz)                 # scalar
    return dw, db

# ── Timing Benchmark ──
m_test = 50000
n_test = 100
X_test = np.random.randn(n_test, m_test)
Y_test = np.random.randint(0, 2, (1, m_test)).astype(float)
w_test = np.random.randn(n_test, 1) * 0.01
b_test = 0.0

# Time the loop version
t1 = time.time()
dw_loop, db_loop = compute_gradients_loop(X_test, Y_test, w_test, b_test)
t_loop = time.time() - t1

# Time the vectorized version
t2 = time.time()
dw_vec, db_vec = compute_gradients_vectorized(X_test, Y_test, w_test, b_test)
t_vec = time.time() - t2

print(f"Loop-based:  {t_loop*1000:.1f} ms")
print(f"Vectorized:  {t_vec*1000:.1f} ms")
print(f"Speedup:     {t_loop/t_vec:.0f}x faster!")
print(f"Results match: {np.allclose(dw_loop, dw_vec)}")

6e. PyTorch Library Version (for comparison)

Python
import torch
import torch.nn as nn

# Convert data to PyTorch tensors
X_pt = torch.tensor(X.T, dtype=torch.float32)  # (m, n)
Y_pt = torch.tensor(Y.T, dtype=torch.float32)  # (m, 1)

# Define the model: logistic regression = 1 Linear layer + Sigmoid
model_pt = nn.Sequential(
    nn.Linear(2, 1),   # z = wᵀx + b
    nn.Sigmoid()          # ŷ = σ(z)
)

# BCE Loss + SGD optimizer
criterion = nn.BCELoss()
optimizer = torch.optim.SGD(model_pt.parameters(), lr=1.0)

# Training loop
for epoch in range(2000):
    y_pred = model_pt(X_pt)
    loss = criterion(y_pred, Y_pt)

    optimizer.zero_grad()
    loss.backward()       # autograd computes gradients automatically!
    optimizer.step()

    if epoch % 500 == 0:
        print(f"Epoch {epoch:4d} | Loss: {loss.item():.6f}")

# Compare: PyTorch did the same forward-backward-update loop,
# but computed gradients automatically via loss.backward().
# Our from-scratch version computed them manually — same math!

def train_step(X, Y, w, b, lr):
    m = X.shape[1]
    z = np.dot(w.T, X) + b
    y_hat = 1 / (1 + np.exp(-z))
    loss = -(1/m) * np.sum(Y * np.log(y_hat) + (1-Y) * np.log(1-y_hat))
    dz = y_hat - Y
    dw = (1/m) * np.dot(X, dz.T)
    db = (1/m) * np.sum(dz)
    w = w + lr * dw      # <-- BUG IS HERE
    b = b + lr * db      # <-- AND HERE
    return w, b, loss

Visual Aids — ASCII Diagrams

Sigmoid Function Shape

BCE Loss Behavior

Full Forward-Backward Dataflow

Common Misconceptions

GATE / Exam Corner

Key Formulas — Quick Reference Sheet

GATE Previous Year Questions (PYQs) Pattern

GATE Prediction Table — What to Expect

Interview Prep — India + US Focus

Conceptual Questions

Coding Interview Questions

# Hint: Use the identity σ(z) = 0.5 * (1 + tanh(z/2))
def sigmoid_via_tanh(z):
    return 0.5 * (1 + np.tanh(z / 2))

def bce_loss(y, y_hat, eps=1e-7):
    y_hat = np.clip(y_hat, eps, 1 - eps)  # prevent log(0)
    return -np.mean(y * np.log(y_hat) + (1 - y) * np.log(1 - y_hat))

Hands-On Lab — SBI Loan Default Predictor

🧪 Mini-Project: Build a Complete Loan Default Prediction System

Exercises — 22 Problems Across 5 Categories

Section A: Conceptual Questions (5)

Section B: Mathematical Problems (8)

Section C: Coding Problems (4)

Section D: Critical Thinking (3)

★ Section E: Starred Research Problems (2)

Connections — Where This Fits

Chapter Summary — 7 Key Takeaways

Further Reading

— End of Chapter 3 —

Next: Chapter 4 → Shallow Neural Networks — What happens when you stack multiple neurons into layers?