Artificial Neurons
& Perceptrons

Learning Objectives

After completing this chapter, you will be able to:

Introduction

The human brain — weighing roughly 1.4 kg — contains approximately 86 billion neurons, each connected to an average of 7,000 other neurons through roughly 150 trillion synapses. This astonishing biological network enables us to see, speak, think, and learn. The question that haunted scientists for nearly a century was: can we build a machine that learns the way neurons do?

In 1943, neurophysiologist Warren McCulloch and logician Walter Pitts published a landmark paper proposing the first mathematical model of a neuron. Fifteen years later, Frank Rosenblatt built the Mark I Perceptron — a physical machine that could learn to classify simple patterns. For a brief, electric moment, the world believed machines would soon rival the human brain.

Then came the XOR problem. In 1969, Marvin Minsky and Seymour Papert published Perceptrons, proving that single-layer perceptrons have fundamental limitations. Funding dried up. The first AI Winter began. Yet the ideas born in this era — weighted sums, activation functions, learning rules — became the foundation of every modern neural network, from GPT to AlphaFold.

This chapter takes you on that journey. We start with biological neurons, build up to mathematical models, implement them in code, and understand both their power and their limits. By the end, you'll have working perceptron and Adaline implementations, and deep intuition for why the XOR problem mattered so much.

Historical Background

The story of artificial neurons spans nearly a century. Here are the pivotal moments:

Conceptual Explanation: The Biological Neuron

Structure of a Biological Neuron

A neuron (nerve cell) is the fundamental unit of the nervous system. Every thought, movement, and sensation arises from networks of neurons communicating through electrical and chemical signals. Let's examine its four key components:

From Biology to Mathematics

The key insight that bridges biology and computation is this mapping:

McCulloch-Pitts Neuron (1943)

Warren McCulloch and Walter Pitts proposed the first mathematical model of a neuron. Their model, called the M-P neuron, is a binary threshold logic unit with the following properties:

Computing Logic Gates with M-P Neurons

AND Gate

The AND gate outputs 1 only when BOTH inputs are 1. We set weights w₁ = w₂ = 1 and threshold θ = 2:

OR Gate

The OR gate outputs 1 when AT LEAST one input is 1. Set w₁ = w₂ = 1 and threshold θ = 1:

NOT Gate

The NOT gate inverts a single input. Set w₁ = −1 and threshold θ = 0 (equivalently: w₁ = −1, bias b = 0.5, threshold = 0):

Alternative formulation: Use w₁ = −1, bias = +0.5. Then z = −0 + 0.5 = 0.5 ≥ 0 → 1, and z = −1 + 0.5 = −0.5 < 0 → 0.

Python: M-P Neuron Implementation

Python
import numpy as np

class McCullochPittsNeuron:
    """McCulloch-Pitts Binary Threshold Neuron (1943)"""

    def __init__(self, weights, threshold):
        """
        Parameters:
        -----------
        weights : list or np.array - fixed weights for each input
        threshold : float - firing threshold θ
        """
        self.weights = np.array(weights, dtype=float)
        self.threshold = threshold

    def activate(self, inputs):
        """Compute output for given binary inputs"""
        x = np.array(inputs, dtype=float)
        z = np.dot(self.weights, x)   # weighted sum
        return 1 if z >= self.threshold else 0

    def truth_table(self, n_inputs):
        """Generate complete truth table for n binary inputs"""
        from itertools import product
        print(f"{'Inputs':<15} {'Sum':>5} {'Output':>7}")
        print("-" * 30)
        for combo in product([0, 1], repeat=n_inputs):
            output = self.activate(combo)
            z = np.dot(self.weights, combo)
            print(f"{str(combo):<15} {z:>5.1f} {output:>7}")

# === AND Gate ===
print("=== AND Gate ===")
and_gate = McCullochPittsNeuron(weights=[1, 1], threshold=2)
and_gate.truth_table(2)

# === OR Gate ===
print("\n=== OR Gate ===")
or_gate = McCullochPittsNeuron(weights=[1, 1], threshold=1)
or_gate.truth_table(2)

# === NOT Gate ===
print("\n=== NOT Gate ===")
not_gate = McCullochPittsNeuron(weights=[-1], threshold=0)
not_gate.truth_table(1)

# === NAND Gate ===
print("\n=== NAND Gate ===")
nand_gate = McCullochPittsNeuron(weights=[-1, -1], threshold=-1)
nand_gate.truth_table(2)

Hebb's Learning Rule (1949)

In 1949, Canadian psychologist Donald Hebb proposed a simple but profound learning principle in his book The Organization of Behavior:

In simpler terms: "Neurons that fire together, wire together." If input neuron A and output neuron B are both active at the same time, strengthen the connection (weight) between them.

Mathematical Formulation

Derivation from First Principles

Step 1: Assume we want the connection weight to grow when both neurons are active simultaneously. The simplest mathematical expression for "both active" is the product xᵢ · yⱼ:

• If xᵢ = 1 and yⱼ = 1 → product = 1 → increase weight
• If xᵢ = 0 or yⱼ = 0 → product = 0 → no change

Step 2: Add a learning rate η to control the magnitude of updates. Too large and weights explode; too small and learning is slow:

Limitation: Hebb's rule only strengthens weights — it never weakens them. Over time, all weights grow unboundedly. This is why pure Hebbian learning is unstable and was later refined by Oja's rule, covariance rules, and the perceptron learning rule.

Hebbian Learning Example

Train an AND gate using Hebb's rule with η = 1, initial weights w₁ = w₂ = 0, bias = 0:

Final weights: w₁ = 1, w₂ = 1. With threshold θ = 2, this correctly computes AND!

Python
import numpy as np

def hebbian_learning(X, y, eta=1.0, epochs=1):
    """
    Hebbian Learning Rule Implementation
    
    Parameters:
    -----------
    X : np.array of shape (n_samples, n_features)
    y : np.array of shape (n_samples,) - target outputs
    eta : float - learning rate
    epochs : int - number of training passes
    
    Returns:
    --------
    weights : learned weights
    """
    n_features = X.shape[1]
    weights = np.zeros(n_features)
    bias = 0.0

    for epoch in range(epochs):
        print(f"\n--- Epoch {epoch + 1} ---")
        for i in range(len(X)):
            xi = X[i]
            yi = y[i]
            
            # Hebb's rule: Δw = η * x * y
            delta_w = eta * xi * yi
            delta_b = eta * yi
            
            weights += delta_w
            bias += delta_b
            
            print(f"  Input: {xi}, Target: {yi}, "
                  f"Δw: {delta_w}, w: {weights}, b: {bias:.1f}")
    
    return weights, bias

# AND gate training data
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 0, 0, 1])

weights, bias = hebbian_learning(X, y, eta=1.0, epochs=1)
print(f"\nFinal weights: {weights}, bias: {bias}")

Rosenblatt's Perceptron (1958)

Frank Rosenblatt's perceptron was the first model that could automatically learn its weights from data. Unlike the M-P neuron (fixed weights), the perceptron adjusts its weights based on errors it makes. This was revolutionary.

Architecture

The Perceptron Learning Algorithm

The genius of the perceptron is its error-driven learning rule:

Why Does This Rule Work?

Let's trace through the three possible cases:

• Case 1: Correct prediction (ŷ = y) → error = 0 → no update needed ✓
• Case 2: False Negative (ŷ = 0, y = 1) → error = +1 → w += α·x → weights increase, making the weighted sum larger, pushing output toward 1 ✓
• Case 3: False Positive (ŷ = 1, y = 0) → error = −1 → w -= α·x → weights decrease, making the weighted sum smaller, pushing output toward 0 ✓

Perceptron Convergence Theorem

Mathematical Foundation

Vector Formulation

The perceptron can be expressed elegantly using vectors and dot products:

Decision Boundary as a Hyperplane

The perceptron's decision boundary is the set of points where z = 0:

In general, a perceptron with n inputs creates an (n-1)-dimensional hyperplane that divides the n-dimensional input space into two half-spaces (classes).

Geometric Interpretation

The weight vector w⃗ is perpendicular (normal) to the decision boundary. The sign of w⃗ · x⃗ + b tells you on which side of the boundary a point lies:

• w⃗ · x⃗ + b > 0 → point is on the positive side → class 1
• w⃗ · x⃗ + b < 0 → point is on the negative side → class 0

Distance from a Point to the Decision Boundary

This distance concept becomes crucial when we study Support Vector Machines (Ch 12) — where we maximize this margin.

Formula Derivations

Deriving the Perceptron Update Rule from First Principles

We want a rule that adjusts weights to reduce classification errors. Let's derive it step by step.

Step 1: Define the goal. We want the perceptron to output ŷ = y for all training samples. When it makes a mistake (ŷ ≠ y), we need to adjust weights.

Step 2: Define the error signal.

The error can be −1, 0, or +1.

Step 3: Determine the direction of adjustment.

If e = +1 (y = 1, ŷ = 0): We need to increase z = w⃗ · x⃗ + b. Since z = Σ wᵢxᵢ + b, increasing wᵢ by an amount proportional to xᵢ will increase z (because Δz = Δwᵢ · xᵢ, and both Δwᵢ and xᵢ should be positive for active inputs).

If e = −1 (y = 0, ŷ = 1): We need to decrease z. Decreasing wᵢ proportional to xᵢ achieves this.

Step 4: Combine into a single rule.

This elegant formula handles all three cases (correct, false negative, false positive) automatically.

Step 5: The learning rate α. We add α ∈ (0, 1] to control the step size. Too large → oscillation; too small → slow convergence.

Deriving the Adaline Cost Function Gradient

Adaline uses the Mean Squared Error (MSE) cost function. Let's derive its gradient from first principles.

Step 1: Define the cost function.

Step 2: Expand for a single sample.

Step 3: Apply the chain rule for ∂J/∂wⱼ.

Step 4: Gradient descent update (moving AGAINST the gradient).

This looks identical to the perceptron rule — but there's a crucial difference: Adaline uses the linear output z (before thresholding), while the perceptron uses the thresholded output ŷ. This means Adaline's cost surface is smooth and convex, enabling true gradient descent.

Worked Numerical Examples

Example 1: AND Gate Perceptron — 3 Training Epochs

Train a perceptron to learn the AND function. Initial: w₁ = 0, w₂ = 0, b = 0, α = 1.

Verification with final weights (w₁=2, w₂=1, b=−2):

Still one error after 3 epochs. The perceptron will continue adjusting and will eventually converge (since AND is linearly separable). Typically converges within 5-7 epochs for AND with α = 1.

Example 2: Adaline Gradient Computation

Given: w₁ = 0.5, w₂ = −0.3, b = 0.1, α = 0.1. Training point: x = (2, 3), y = 1.

The XOR Problem

The XOR (exclusive OR) problem is arguably the most important problem in the history of neural networks. It demonstrated a fundamental limitation of single-layer perceptrons and sparked the first AI Winter.

XOR Truth Table

Proof 1: Geometric Impossibility

A single perceptron's decision boundary is a straight line. Let's plot the XOR points:

Points of the same class (y=1) are at diagonally opposite corners (0,1) and (1,0). Points of class y=0 are at (0,0) and (1,1). No single straight line can separate them.

Proof 2: Algebraic Impossibility

Multi-Layer Solution to XOR

XOR can be decomposed as: XOR(x₁, x₂) = AND(OR(x₁, x₂), NAND(x₁, x₂))

Python
import numpy as np

def step(z):
    """Step activation function"""
    return 1 if z >= 0 else 0

def xor_network(x1, x2):
    """
    Multi-layer perceptron solving XOR
    Layer 1: OR + NAND neurons
    Layer 2: AND neuron
    """
    # Hidden Layer
    # OR neuron: w1=1, w2=1, b=-0.5 (threshold=0.5)
    h1 = step(1*x1 + 1*x2 - 0.5)    # OR gate

    # NAND neuron: w1=-1, w2=-1, b=1.5
    h2 = step(-1*x1 + -1*x2 + 1.5)  # NAND gate

    # Output Layer
    # AND neuron: w1=1, w2=1, b=-1.5
    y = step(1*h1 + 1*h2 - 1.5)     # AND gate

    return y

# Verify XOR
print("XOR Network Verification:")
print(f"XOR(0,0) = {xor_network(0,0)}")  # 0
print(f"XOR(0,1) = {xor_network(0,1)}")  # 1
print(f"XOR(1,0) = {xor_network(1,0)}")  # 1
print(f"XOR(1,1) = {xor_network(1,1)}")  # 0

Activation Functions

Activation functions introduce non-linearity into neural networks. Without them, any network — no matter how deep — would be equivalent to a single linear transformation. Let's study the four foundational activation functions.

1. Step Function (Heaviside)

Problem: The derivative is zero everywhere, so gradient-based learning (backpropagation) cannot work. Used in perceptrons but NOT in modern deep learning.

2. Sigmoid (Logistic)

Pros: Smooth, differentiable, outputs in (0,1) — great for probability. Cons: Vanishing gradients for |z| >> 0 (σ'(z) → 0), not zero-centered, slow convergence.

3. Tanh (Hyperbolic Tangent)

Advantage over sigmoid: Zero-centered (outputs between −1 and 1), which helps gradient descent converge faster.

4. ReLU (Rectified Linear Unit)

Pros: Computationally fast, no vanishing gradient for z > 0, sparse activations. Cons: "Dying ReLU" problem (neurons with z < 0 always output 0, never recover).

Summary Comparison

Python
import numpy as np
import matplotlib.pyplot as plt

z = np.linspace(-5, 5, 200)

# Activation functions
def step(z): return np.where(z >= 0, 1, 0)
def sigmoid(z): return 1 / (1 + np.exp(-z))
def tanh_fn(z): return np.tanh(z)
def relu(z): return np.maximum(0, z)

# Derivatives
def sigmoid_deriv(z): s = sigmoid(z); return s * (1 - s)
def tanh_deriv(z): return 1 - np.tanh(z)**2
def relu_deriv(z): return np.where(z > 0, 1, 0)

fig, axes = plt.subplots(2, 4, figsize=(16, 6))

# Functions
for ax, fn, name, color in zip(
    axes[0],
    [step, sigmoid, tanh_fn, relu],
    ['Step', 'Sigmoid', 'Tanh', 'ReLU'],
    ['#ef4444', '#3b82f6', '#a855f7', '#10b981']
):
    ax.plot(z, fn(z), color=color, linewidth=2.5)
    ax.set_title(name, fontsize=12, fontweight='bold')
    ax.axhline(y=0, color='gray', linewidth=0.5)
    ax.axvline(x=0, color='gray', linewidth=0.5)
    ax.grid(True, alpha=0.2)
    ax.set_xlabel('z')

# Derivatives
for ax, fn, name, color in zip(
    axes[1],
    [lambda z: np.zeros_like(z), sigmoid_deriv, tanh_deriv, relu_deriv],
    ["Step'", "Sigmoid'", "Tanh'", "ReLU'"],
    ['#ef4444', '#3b82f6', '#a855f7', '#10b981']
):
    ax.plot(z, fn(z), color=color, linewidth=2.5, linestyle='--')
    ax.set_title(f'{name} (Derivative)', fontsize=12, fontweight='bold')
    ax.axhline(y=0, color='gray', linewidth=0.5)
    ax.axvline(x=0, color='gray', linewidth=0.5)
    ax.grid(True, alpha=0.2)
    ax.set_xlabel('z')

plt.suptitle('Activation Functions & Their Derivatives', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.savefig('activation_functions.png', dpi=150, bbox_inches='tight')
plt.show()

Adaline (Adaptive Linear Neuron)

In 1960, Bernard Widrow and Ted Hoff at Stanford introduced Adaline — a critical improvement over the perceptron. The key difference: Adaline computes its error before the threshold function, using the continuous linear output.

Adaline vs. Perceptron

The MSE Cost Function

This cost function is a convex paraboloid — it has a single global minimum. Gradient descent is guaranteed to find it (with appropriate learning rate).

Gradient Descent Update (Batch)

Adaline vs. Perceptron: Key Differences

Visual Diagrams

Complete Perceptron Architecture

Decision Boundaries for Logic Gates

Linearly Separable vs. Non-Separable Data

Flowcharts

Perceptron Learning Algorithm Flowchart

Adaline vs Perceptron — Comparison Flowchart

Python Implementation from Scratch

Perceptron Class

Python
import numpy as np
import matplotlib.pyplot as plt

class Perceptron:
    """
    Single-Layer Perceptron Classifier
    
    Implements Rosenblatt's Perceptron (1958) with the
    original error-driven learning rule.
    
    Parameters
    ----------
    learning_rate : float (default=0.01)
        Learning rate (α) for weight updates.
    n_epochs : int (default=100)
        Maximum number of training epochs.
    random_state : int (default=42)
        Seed for reproducible weight initialization.
    
    Attributes
    ----------
    weights_ : np.array of shape (n_features,)
        Learned weights after fitting.
    bias_ : float
        Learned bias term.
    errors_ : list
        Number of misclassifications in each epoch.
    """
    
    def __init__(self, learning_rate=0.01, n_epochs=100, random_state=42):
        self.learning_rate = learning_rate
        self.n_epochs = n_epochs
        self.random_state = random_state
    
    def fit(self, X, y):
        """
        Train the perceptron on labeled data.
        
        Parameters
        ----------
        X : np.array of shape (n_samples, n_features)
        y : np.array of shape (n_samples,) - binary labels {0, 1}
        
        Returns
        -------
        self : fitted perceptron
        """
        rng = np.random.RandomState(self.random_state)
        self.weights_ = rng.normal(loc=0.0, scale=0.01,
                                    size=X.shape[1])
        self.bias_ = 0.0
        self.errors_ = []
        
        for epoch in range(self.n_epochs):
            errors = 0
            for xi, yi in zip(X, y):
                # Step 1: Compute prediction
                y_pred = self.predict_single(xi)
                
                # Step 2: Compute error
                error = yi - y_pred
                
                # Step 3: Update weights and bias
                self.weights_ += self.learning_rate * error * xi
                self.bias_ += self.learning_rate * error
                
                # Count errors
                errors += int(error != 0)
            
            self.errors_.append(errors)
            
            # Early stopping if converged
            if errors == 0:
                print(f"Converged at epoch {epoch + 1}")
                break
        
        return self
    
    def net_input(self, X):
        """Compute weighted sum z = w·x + b"""
        return np.dot(X, self.weights_) + self.bias_
    
    def predict_single(self, x):
        """Predict class for a single sample"""
        return 1 if self.net_input(x) >= 0.0 else 0
    
    def predict(self, X):
        """Predict class labels for multiple samples"""
        return np.where(self.net_input(X) >= 0.0, 1, 0)
    
    def accuracy(self, X, y):
        """Compute classification accuracy"""
        predictions = self.predict(X)
        return np.mean(predictions == y)
    
    def plot_errors(self):
        """Plot number of errors per epoch"""
        plt.figure(figsize=(8, 4))
        plt.plot(range(1, len(self.errors_) + 1),
                 self.errors_, marker='o', color='#059669')
        plt.xlabel('Epoch')
        plt.ylabel('Number of Misclassifications')
        plt.title('Perceptron Training - Errors per Epoch')
        plt.grid(True, alpha=0.3)
        plt.tight_layout()
        plt.show()
    
    def plot_decision_boundary(self, X, y, title="Perceptron Decision Boundary"):
        """Plot 2D decision boundary"""
        if X.shape[1] != 2:
            raise ValueError("Can only plot 2D decision boundaries")
        
        # Create mesh
        x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
        y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
        xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.02),
                              np.arange(y_min, y_max, 0.02))
        Z = self.predict(np.c_[xx.ravel(), yy.ravel()])
        Z = Z.reshape(xx.shape)
        
        plt.figure(figsize=(8, 6))
        plt.contourf(xx, yy, Z, alpha=0.3,
                     cmap=plt.cm.RdYlGn)
        plt.scatter(X[y == 0][:, 0], X[y == 0][:, 1],
                   c='red', marker='o', label='Class 0',
                   edgecolors='k', s=80)
        plt.scatter(X[y == 1][:, 0], X[y == 1][:, 1],
                   c='green', marker='^', label='Class 1',
                   edgecolors='k', s=80)
        plt.xlabel('x₁')
        plt.ylabel('x₂')
        plt.title(title)
        plt.legend()
        plt.grid(True, alpha=0.3)
        plt.tight_layout()
        plt.show()


# ============ Demo: Logic Gates ============
print("=" * 50)
print("PERCEPTRON LOGIC GATE LEARNING")
print("=" * 50)

# AND gate
X_and = np.array([[0,0], [0,1], [1,0], [1,1]])
y_and = np.array([0, 0, 0, 1])

p_and = Perceptron(learning_rate=0.1, n_epochs=20)
p_and.fit(X_and, y_and)
print(f"\nAND Gate - Weights: {p_and.weights_}, Bias: {p_and.bias_:.2f}")
print(f"Predictions: {p_and.predict(X_and)}")
print(f"Accuracy: {p_and.accuracy(X_and, y_and):.0%}")

# OR gate
y_or = np.array([0, 1, 1, 1])
p_or = Perceptron(learning_rate=0.1, n_epochs=20)
p_or.fit(X_and, y_or)
print(f"\nOR Gate  - Weights: {p_or.weights_}, Bias: {p_or.bias_:.2f}")
print(f"Predictions: {p_or.predict(X_and)}")

# XOR gate (will NOT converge!)
y_xor = np.array([0, 1, 1, 0])
p_xor = Perceptron(learning_rate=0.1, n_epochs=20)
p_xor.fit(X_and, y_xor)
print(f"\nXOR Gate - Weights: {p_xor.weights_}, Bias: {p_xor.bias_:.2f}")
print(f"Predictions: {p_xor.predict(X_and)}")
print(f"Accuracy: {p_xor.accuracy(X_and, y_xor):.0%}")
print("XOR did NOT converge — as expected!")

Adaline Class

Python
import numpy as np
import matplotlib.pyplot as plt

class Adaline:
    """
    ADAptive LInear NEuron (Adaline) classifier.
    
    Uses gradient descent on the Mean Squared Error (MSE) cost
    function to learn weights. Error is computed on the LINEAR
    output (before thresholding), unlike the perceptron.
    
    Parameters
    ----------
    learning_rate : float (default=0.01)
        Learning rate α for gradient descent.
    n_epochs : int (default=100)
        Number of training epochs.
    random_state : int (default=42)
        Seed for weight initialization.
    
    Attributes
    ----------
    weights_ : np.array - learned weights
    bias_ : float - learned bias
    cost_ : list - MSE cost at each epoch
    """
    
    def __init__(self, learning_rate=0.01, n_epochs=100, random_state=42):
        self.learning_rate = learning_rate
        self.n_epochs = n_epochs
        self.random_state = random_state
    
    def fit(self, X, y):
        """
        Train Adaline using batch gradient descent.
        
        Parameters
        ----------
        X : np.array of shape (n_samples, n_features)
        y : np.array of shape (n_samples,)
        """
        rng = np.random.RandomState(self.random_state)
        self.weights_ = rng.normal(loc=0.0, scale=0.01,
                                    size=X.shape[1])
        self.bias_ = 0.0
        self.cost_ = []
        
        for epoch in range(self.n_epochs):
            # Forward pass: compute linear activation
            z = self.net_input(X)           # z = Xw + b
            
            # Compute errors (continuous, not thresholded)
            errors = y - z                   # e = y - z
            
            # Batch gradient descent update
            # ∂J/∂w = -(1/N) * Xᵀ(y - z)
            self.weights_ += self.learning_rate * (1.0 / len(y)) * X.T.dot(errors)
            self.bias_ += self.learning_rate * errors.mean()
            
            # Compute cost (MSE)
            cost = 0.5 * np.mean(errors ** 2)
            self.cost_.append(cost)
        
        return self
    
    def net_input(self, X):
        """Compute linear activation z = Xw + b"""
        return np.dot(X, self.weights_) + self.bias_
    
    def predict(self, X):
        """Predict class labels using thresholded output"""
        return np.where(self.net_input(X) >= 0.5, 1, 0)
    
    def accuracy(self, X, y):
        return np.mean(self.predict(X) == y)
    
    def plot_cost(self):
        """Plot MSE cost over epochs"""
        plt.figure(figsize=(8, 4))
        plt.plot(range(1, len(self.cost_) + 1),
                 self.cost_, marker='.', color='#0891b2')
        plt.xlabel('Epoch')
        plt.ylabel('Mean Squared Error')
        plt.title('Adaline Training - Cost per Epoch')
        plt.grid(True, alpha=0.3)
        plt.yscale('log')
        plt.tight_layout()
        plt.show()
    
    def plot_decision_boundary(self, X, y, title="Adaline Decision Boundary"):
        """Plot 2D decision boundary"""
        x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
        y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
        xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.02),
                              np.arange(y_min, y_max, 0.02))
        Z = self.predict(np.c_[xx.ravel(), yy.ravel()])
        Z = Z.reshape(xx.shape)
        
        plt.figure(figsize=(8, 6))
        plt.contourf(xx, yy, Z, alpha=0.3, cmap=plt.cm.RdYlGn)
        plt.scatter(X[y == 0][:, 0], X[y == 0][:, 1],
                   c='red', marker='o', label='Class 0',
                   edgecolors='k', s=80)
        plt.scatter(X[y == 1][:, 0], X[y == 1][:, 1],
                   c='green', marker='^', label='Class 1',
                   edgecolors='k', s=80)
        plt.xlabel('x₁')
        plt.ylabel('x₂')
        plt.title(title)
        plt.legend()
        plt.grid(True, alpha=0.3)
        plt.tight_layout()
        plt.show()


# ============ Demo: Adaline on Iris Dataset ============
from sklearn.datasets import load_iris

# Load Iris (first 2 classes, first 2 features)
iris = load_iris()
X_iris = iris.data[:100, :2]  # Setosa vs Versicolor
y_iris = iris.target[:100]     # 0 and 1

# Standardize features (important for Adaline!)
X_std = (X_iris - X_iris.mean(axis=0)) / X_iris.std(axis=0)

# Train Adaline
ada = Adaline(learning_rate=0.01, n_epochs=50)
ada.fit(X_std, y_iris)

print(f"Adaline Accuracy: {ada.accuracy(X_std, y_iris):.0%}")
print(f"Final weights: {ada.weights_}")
print(f"Final bias: {ada.bias_:.4f}")
print(f"Final MSE: {ada.cost_[-1]:.6f}")

# Plot cost convergence
ada.plot_cost()

# Plot decision boundary
ada.plot_decision_boundary(X_std, y_iris,
                           title="Adaline on Iris (Standardized)")

# ============ Effect of Learning Rate ============
fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# Too large learning rate
ada_large = Adaline(learning_rate=0.1, n_epochs=50).fit(X_std, y_iris)
axes[0].plot(range(1, 51), ada_large.cost_, color='red')
axes[0].set_title('α = 0.1 (Too Large - Diverges!)')
axes[0].set_xlabel('Epoch')
axes[0].set_ylabel('MSE')

# Good learning rate
ada_good = Adaline(learning_rate=0.01, n_epochs=50).fit(X_std, y_iris)
axes[1].plot(range(1, 51), ada_good.cost_, color='green')
axes[1].set_title('α = 0.01 (Good - Converges)')
axes[1].set_xlabel('Epoch')
axes[1].set_ylabel('MSE')

plt.suptitle('Effect of Learning Rate on Adaline Convergence')
plt.tight_layout()
plt.show()

TensorFlow Implementation

TensorFlow's Dense layer with a single unit is essentially a perceptron/Adaline. Here we build single-neuron models using TensorFlow.

TensorFlow
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

# ============================================================
# 1. Single Neuron as a Binary Classifier (like Perceptron)
# ============================================================

# Load Iris data (2 classes)
iris = load_iris()
X = iris.data[:100]     # Setosa vs Versicolor (100 samples)
y = iris.target[:100]   # Binary labels: 0 or 1

# Split and standardize
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)

# Build a single-neuron model (1 Dense unit with sigmoid)
model = keras.Sequential([
    layers.Input(shape=(4,)),
    layers.Dense(1, activation='sigmoid',
                 kernel_initializer='zeros',
                 bias_initializer='zeros',
                 name='single_neuron')
])

model.compile(
    optimizer=keras.optimizers.SGD(learning_rate=0.1),
    loss='binary_crossentropy',
    metrics=['accuracy']
)

model.summary()

# Train
history = model.fit(
    X_train_s, y_train,
    epochs=50,
    batch_size=16,
    validation_data=(X_test_s, y_test),
    verbose=1
)

# Evaluate
loss, acc = model.evaluate(X_test_s, y_test, verbose=0)
print(f"\nTest Accuracy: {acc:.2%}")
print(f"Test Loss: {loss:.4f}")

# Get learned weights
weights, bias = model.get_layer('single_neuron').get_weights()
print(f"\nLearned Weights: {weights.flatten()}")
print(f"Learned Bias: {bias[0]:.4f}")

# Plot training history
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

ax1.plot(history.history['loss'], label='Train Loss')
ax1.plot(history.history['val_loss'], label='Val Loss')
ax1.set_xlabel('Epoch')
ax1.set_ylabel('Loss')
ax1.set_title('Training & Validation Loss')
ax1.legend()
ax1.grid(True, alpha=0.3)

ax2.plot(history.history['accuracy'], label='Train Acc')
ax2.plot(history.history['val_accuracy'], label='Val Acc')
ax2.set_xlabel('Epoch')
ax2.set_ylabel('Accuracy')
ax2.set_title('Training & Validation Accuracy')
ax2.legend()
ax2.grid(True, alpha=0.3)

plt.suptitle('TensorFlow Single Neuron Training', fontsize=14)
plt.tight_layout()
plt.show()


# ============================================================
# 2. XOR with Multi-Layer Perceptron in TensorFlow
# ============================================================

X_xor = np.array([[0,0], [0,1], [1,0], [1,1]], dtype=np.float32)
y_xor = np.array([0, 1, 1, 0], dtype=np.float32)

# Single neuron (will fail!)
model_single = keras.Sequential([
    layers.Input(shape=(2,)),
    layers.Dense(1, activation='sigmoid')
])
model_single.compile(optimizer='sgd', loss='binary_crossentropy',
                     metrics=['accuracy'])
model_single.fit(X_xor, y_xor, epochs=1000, verbose=0)
print("\n--- XOR with Single Neuron ---")
preds = (model_single.predict(X_xor, verbose=0) > 0.5).astype(int)
print(f"Predictions: {preds.flatten()} (Expected: [0, 1, 1, 0])")
print(f"Accuracy: {np.mean(preds.flatten() == y_xor):.0%}")

# Multi-layer (will succeed!)
model_mlp = keras.Sequential([
    layers.Input(shape=(2,)),
    layers.Dense(4, activation='relu'),    # Hidden layer
    layers.Dense(1, activation='sigmoid')  # Output layer
])
model_mlp.compile(optimizer=keras.optimizers.Adam(0.1),
                  loss='binary_crossentropy',
                  metrics=['accuracy'])
model_mlp.fit(X_xor, y_xor, epochs=500, verbose=0)
print("\n--- XOR with Multi-Layer Perceptron ---")
preds_mlp = (model_mlp.predict(X_xor, verbose=0) > 0.5).astype(int)
print(f"Predictions: {preds_mlp.flatten()} (Expected: [0, 1, 1, 0])")
print(f"Accuracy: {np.mean(preds_mlp.flatten() == y_xor):.0%}")

Scikit-Learn Implementation

Scikit-Learn
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import Perceptron, SGDClassifier
from sklearn.datasets import load_iris, make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (classification_report, confusion_matrix,
                             accuracy_score)

# ============================================================
# 1. sklearn.linear_model.Perceptron
# ============================================================

# Load Iris (binary: Setosa vs Versicolor)
iris = load_iris()
X = iris.data[:100]
y = iris.target[:100]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

# Standardize
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)

# Train Perceptron
clf = Perceptron(
    eta0=0.1,            # learning rate
    max_iter=100,        # max epochs
    tol=1e-3,            # convergence tolerance
    random_state=42,
    verbose=1
)
clf.fit(X_train_s, y_train)

# Results
y_pred = clf.predict(X_test_s)
print(f"\n{'='*50}")
print(f"Sklearn Perceptron Results")
print(f"{'='*50}")
print(f"Accuracy: {accuracy_score(y_test, y_pred):.2%}")
print(f"Coefficients: {clf.coef_}")
print(f"Intercept: {clf.intercept_}")
print(f"Number of iterations: {clf.n_iter_}")
print(f"\nClassification Report:")
print(classification_report(y_test, y_pred,
      target_names=['Setosa', 'Versicolor']))


# ============================================================
# 2. Cross-Validation for Robust Evaluation
# ============================================================

X_all_s = scaler.fit_transform(X)
cv_scores = cross_val_score(
    Perceptron(eta0=0.1, max_iter=100, random_state=42),
    X_all_s, y, cv=5, scoring='accuracy'
)
print(f"\n5-Fold CV Accuracy: {cv_scores.mean():.2%} ± {cv_scores.std():.2%}")
print(f"Individual folds: {cv_scores}")


# ============================================================
# 3. SGDClassifier with perceptron loss (equivalent)
# ============================================================

sgd_perceptron = SGDClassifier(
    loss='perceptron',     # equivalent to Perceptron class
    eta0=0.1,
    learning_rate='constant',
    max_iter=100,
    random_state=42
)
sgd_perceptron.fit(X_train_s, y_train)
print(f"\nSGDClassifier (perceptron loss) Accuracy: "
      f"{sgd_perceptron.score(X_test_s, y_test):.2%}")


# ============================================================
# 4. Larger Dataset: Make Classification
# ============================================================

X_large, y_large = make_classification(
    n_samples=1000,
    n_features=10,
    n_informative=5,
    n_redundant=2,
    n_classes=2,
    random_state=42
)

X_tr, X_te, y_tr, y_te = train_test_split(
    X_large, y_large, test_size=0.3, random_state=42
)

sc = StandardScaler()
X_tr_s = sc.fit_transform(X_tr)
X_te_s = sc.transform(X_te)

perc = Perceptron(eta0=0.01, max_iter=200, random_state=42)
perc.fit(X_tr_s, y_tr)

print(f"\nLarger Dataset (1000 samples, 10 features):")
print(f"Train Accuracy: {perc.score(X_tr_s, y_tr):.2%}")
print(f"Test Accuracy: {perc.score(X_te_s, y_te):.2%}")


# ============================================================
# 5. Decision Boundary Visualization (2D projection)
# ============================================================

# Use only first 2 features for visualization
X_2d = iris.data[:100, :2]
X_tr2, X_te2, y_tr2, y_te2 = train_test_split(
    X_2d, y, test_size=0.3, random_state=42
)
sc2 = StandardScaler()
X_tr2_s = sc2.fit_transform(X_tr2)
X_te2_s = sc2.transform(X_te2)

perc2d = Perceptron(eta0=0.1, max_iter=100, random_state=42)
perc2d.fit(X_tr2_s, y_tr2)

# Plot
x_min, x_max = X_tr2_s[:, 0].min() - 1, X_tr2_s[:, 0].max() + 1
y_min, y_max = X_tr2_s[:, 1].min() - 1, X_tr2_s[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.02),
                      np.arange(y_min, y_max, 0.02))
Z = perc2d.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

plt.figure(figsize=(8, 6))
plt.contourf(xx, yy, Z, alpha=0.3, cmap='RdYlGn')
plt.scatter(X_tr2_s[y_tr2==0, 0], X_tr2_s[y_tr2==0, 1],
           c='red', marker='o', label='Setosa (train)', s=60)
plt.scatter(X_tr2_s[y_tr2==1, 0], X_tr2_s[y_tr2==1, 1],
           c='green', marker='^', label='Versicolor (train)', s=60)
plt.scatter(X_te2_s[y_te2==0, 0], X_te2_s[y_te2==0, 1],
           c='red', marker='o', alpha=0.4, label='Setosa (test)', s=40)
plt.scatter(X_te2_s[y_te2==1, 0], X_te2_s[y_te2==1, 1],
           c='green', marker='^', alpha=0.4, label='Versicolor (test)', s=40)
plt.xlabel('Sepal Length (std)')
plt.ylabel('Sepal Width (std)')
plt.title('Sklearn Perceptron - Iris Decision Boundary')
plt.legend(loc='upper left')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

Indian Case Studies

Global Case Studies

Startup, Government & Industry Applications

🚀 Startup Applications

🏛️ Government Applications

🏭 Industry Applications

Mini Projects

Mini Project 1: Logic Gate Learner

Python
import numpy as np
import matplotlib.pyplot as plt

class LogicGateLearner:
    """Interactive Logic Gate Learner using Perceptrons"""

    GATES = {
        'AND':  np.array([0, 0, 0, 1]),
        'OR':   np.array([0, 1, 1, 1]),
        'NAND': np.array([1, 1, 1, 0]),
        'NOR':  np.array([1, 0, 0, 0]),
        'XOR':  np.array([0, 1, 1, 0]),
        'XNOR': np.array([1, 0, 0, 1]),
    }

    X = np.array([[0,0], [0,1], [1,0], [1,1]])

    def __init__(self, gate_name, lr=0.1, max_epochs=100):
        self.gate_name = gate_name.upper()
        self.y = self.GATES[self.gate_name]
        self.lr = lr
        self.max_epochs = max_epochs
        self.weights = np.zeros(2)
        self.bias = 0.0
        self.history = []

    def step(self, z):
        return (z >= 0).astype(int)

    def train_perceptron(self):
        """Train single-layer perceptron"""
        for epoch in range(self.max_epochs):
            errors = 0
            for xi, yi in zip(self.X, self.y):
                z = np.dot(xi, self.weights) + self.bias
                y_pred = 1 if z >= 0 else 0
                error = yi - y_pred
                self.weights += self.lr * error * xi
                self.bias += self.lr * error
                errors += int(error != 0)
            self.history.append(errors)
            if errors == 0:
                return True, epoch + 1
        return False, self.max_epochs

    def train_mlp_xor(self):
        """2-layer solution for XOR/XNOR"""
        results = []
        for xi in self.X:
            h1 = 1 if (xi[0] + xi[1] - 0.5) >= 0 else 0     # OR
            h2 = 1 if (-xi[0] - xi[1] + 1.5) >= 0 else 0     # NAND
            if self.gate_name == 'XOR':
                y = 1 if (h1 + h2 - 1.5) >= 0 else 0          # AND
            else:  # XNOR
                y = 1 if (-h1 - h2 + 0.5) >= 0 else 0         # NOR-like
            results.append(y)
        return np.array(results)

    def run(self):
        print(f"\n{'='*50}")
        print(f"Learning {self.gate_name} Gate")
        print(f"{'='*50}")
        print(f"Truth table: {self.y}")

        converged, epochs = self.train_perceptron()

        if converged:
            preds = self.step(self.X @ self.weights + self.bias)
            print(f"✅ CONVERGED in {epochs} epochs!")
            print(f"   Weights: [{self.weights[0]:.2f}, {self.weights[1]:.2f}]")
            print(f"   Bias: {self.bias:.2f}")
            print(f"   Predictions: {preds}")
            print(f"   {self.gate_name} IS linearly separable.")
        else:
            print(f"❌ DID NOT CONVERGE after {epochs} epochs.")
            print(f"   {self.gate_name} is NOT linearly separable!")
            print(f"   A single perceptron CANNOT learn {self.gate_name}.")

            if self.gate_name in ['XOR', 'XNOR']:
                print(f"\n   Solving with 2-layer network...")
                mlp_preds = self.train_mlp_xor()
                print(f"   MLP Predictions: {mlp_preds}")
                print(f"   Expected:        {self.y}")
                print(f"   ✅ Multi-layer solution works!")

# Run all gates
for gate in ['AND', 'OR', 'NAND', 'NOR', 'XOR', 'XNOR']:
    learner = LogicGateLearner(gate)
    learner.run()

Mini Project 2: Simple Pattern Classifier

Python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (accuracy_score, precision_score,
                             recall_score, f1_score,
                             confusion_matrix, ConfusionMatrixDisplay)

# Load Wisconsin Breast Cancer dataset
data = load_breast_cancer()
X, y = data.data, data.target
feature_names = data.feature_names

print(f"Dataset: {data.DESCR[:200]}...")
print(f"Samples: {X.shape[0]}, Features: {X.shape[1]}")
print(f"Classes: {np.unique(y)} (0=malignant, 1=benign)")
print(f"Class distribution: {np.bincount(y)}")

# Split and standardize
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)

# === Custom Perceptron ===
class PerceptronClassifier:
    def __init__(self, lr=0.01, epochs=100):
        self.lr = lr
        self.epochs = epochs

    def fit(self, X, y):
        self.w = np.zeros(X.shape[1])
        self.b = 0.0
        self.errors = []

        for ep in range(self.epochs):
            err = 0
            for xi, yi in zip(X, y):
                pred = 1 if np.dot(xi, self.w) + self.b >= 0 else 0
                update = self.lr * (yi - pred)
                self.w += update * xi
                self.b += update
                err += int(update != 0)
            self.errors.append(err)
            if err == 0:
                print(f"  Perceptron converged at epoch {ep+1}")
                break
        return self

    def predict(self, X):
        return np.where(np.dot(X, self.w) + self.b >= 0, 1, 0)

# === Custom Adaline ===
class AdalineClassifier:
    def __init__(self, lr=0.001, epochs=100):
        self.lr = lr
        self.epochs = epochs

    def fit(self, X, y):
        self.w = np.zeros(X.shape[1])
        self.b = 0.0
        self.cost = []

        for ep in range(self.epochs):
            z = X @ self.w + self.b
            errors = y - z
            self.w += self.lr * (1/len(y)) * X.T @ errors
            self.b += self.lr * errors.mean()
            mse = 0.5 * np.mean(errors**2)
            self.cost.append(mse)
        return self

    def predict(self, X):
        return np.where(X @ self.w + self.b >= 0.5, 1, 0)

# Train both models
print("\n--- Training Perceptron ---")
perc = PerceptronClassifier(lr=0.01, epochs=100)
perc.fit(X_train_s, y_train)

print("\n--- Training Adaline ---")
ada = AdalineClassifier(lr=0.001, epochs=100)
ada.fit(X_train_s, y_train)

# Evaluate both
for name, model in [("Perceptron", perc), ("Adaline", ada)]:
    y_pred = model.predict(X_test_s)
    print(f"\n{'='*40}")
    print(f"{name} Results")
    print(f"{'='*40}")
    print(f"Accuracy:  {accuracy_score(y_test, y_pred):.2%}")
    print(f"Precision: {precision_score(y_test, y_pred):.2%}")
    print(f"Recall:    {recall_score(y_test, y_pred):.2%}")
    print(f"F1 Score:  {f1_score(y_test, y_pred):.2%}")

# Plot comparison
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))

ax1.plot(perc.errors, color='#059669', linewidth=2)
ax1.set_title('Perceptron: Errors per Epoch', fontweight='bold')
ax1.set_xlabel('Epoch')
ax1.set_ylabel('Errors')
ax1.grid(True, alpha=0.3)

ax2.plot(ada.cost, color='#0891b2', linewidth=2)
ax2.set_title('Adaline: MSE Cost per Epoch', fontweight='bold')
ax2.set_xlabel('Epoch')
ax2.set_ylabel('MSE')
ax2.grid(True, alpha=0.3)

plt.suptitle('Breast Cancer Classification: Perceptron vs Adaline',
             fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()

# Feature importance (based on weight magnitudes)
top_k = 10
perc_importance = np.abs(perc.w)
top_features = np.argsort(perc_importance)[-top_k:]

plt.figure(figsize=(10, 5))
plt.barh(range(top_k),
         perc_importance[top_features],
         color='#059669', alpha=0.8)
plt.yticks(range(top_k),
           [feature_names[i] for i in top_features])
plt.xlabel('|Weight|')
plt.title('Top 10 Most Important Features (Perceptron)')
plt.tight_layout()
plt.show()

Mini Project 3: Activation Function Explorer

Python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.gridspec import GridSpec

class ActivationExplorer:
    """Interactive activation function comparison tool"""
    
    FUNCTIONS = {
        'Step': {
            'fn': lambda z: np.where(z >= 0, 1, 0).astype(float),
            'deriv': lambda z: np.zeros_like(z),
            'color': '#ef4444', 'range': '{0, 1}'
        },
        'Sigmoid': {
            'fn': lambda z: 1 / (1 + np.exp(-np.clip(z, -500, 500))),
            'deriv': lambda z: (1/(1+np.exp(-np.clip(z,-500,500)))) * 
                               (1 - 1/(1+np.exp(-np.clip(z,-500,500)))),
            'color': '#3b82f6', 'range': '(0, 1)'
        },
        'Tanh': {
            'fn': lambda z: np.tanh(z),
            'deriv': lambda z: 1 - np.tanh(z)**2,
            'color': '#a855f7', 'range': '(-1, 1)'
        },
        'ReLU': {
            'fn': lambda z: np.maximum(0, z),
            'deriv': lambda z: np.where(z > 0, 1.0, 0.0),
            'color': '#10b981', 'range': '[0, ∞)'
        },
        'Leaky ReLU': {
            'fn': lambda z: np.where(z > 0, z, 0.01 * z),
            'deriv': lambda z: np.where(z > 0, 1.0, 0.01),
            'color': '#f59e0b', 'range': '(-∞, ∞)'
        },
        'ELU': {
            'fn': lambda z: np.where(z > 0, z, 1.0*(np.exp(z)-1)),
            'deriv': lambda z: np.where(z > 0, 1.0,
                                        1.0*np.exp(z)),
            'color': '#ec4899', 'range': '(-α, ∞)'
        }
    }
    
    def plot_all(self, z_range=(-5, 5)):
        z = np.linspace(*z_range, 500)
        n = len(self.FUNCTIONS)
        
        fig = plt.figure(figsize=(18, 10))
        gs = GridSpec(2, n, figure=fig)
        
        for i, (name, spec) in enumerate(self.FUNCTIONS.items()):
            # Function
            ax_fn = fig.add_subplot(gs[0, i])
            ax_fn.plot(z, spec['fn'](z), color=spec['color'],
                      linewidth=2.5)
            ax_fn.set_title(f'{name}\nRange: {spec["range"]}',
                          fontsize=9, fontweight='bold')
            ax_fn.axhline(0, color='gray', lw=0.5)
            ax_fn.axvline(0, color='gray', lw=0.5)
            ax_fn.grid(True, alpha=0.2)
            ax_fn.set_xlabel('z', fontsize=8)
            
            # Derivative
            ax_d = fig.add_subplot(gs[1, i])
            ax_d.plot(z, spec['deriv'](z), color=spec['color'],
                     linewidth=2.5, linestyle='--')
            ax_d.set_title(f"{name}' (derivative)",
                          fontsize=9, fontweight='bold')
            ax_d.axhline(0, color='gray', lw=0.5)
            ax_d.axvline(0, color='gray', lw=0.5)
            ax_d.grid(True, alpha=0.2)
            ax_d.set_xlabel('z', fontsize=8)
        
        plt.suptitle('Activation Functions & Derivatives Comparison',
                    fontsize=14, fontweight='bold')
        plt.tight_layout()
        plt.savefig('activation_explorer.png', dpi=150, bbox_inches='tight')
        plt.show()

explorer = ActivationExplorer()
explorer.plot_all()

End-of-Chapter Exercises

Conceptual Exercises

Mathematical Exercises

Programming Exercises

Advanced Exercises

Multiple Choice Questions (MCQs)

Interview Questions

Research Problems

Key Takeaways

References