Chapter 11: Hyperparameter Tuning & ML Strategy

Bloom's Taxonomy Map for This Chapter

Learning Objectives

By the end of this chapter, you will be able to:

• Rank hyperparameters by importance (Tier 1–4) and justify why learning rate is the single most critical hyperparameter
• Compare grid search vs. random search using the probability argument, and explain when random search is strictly superior
• Apply the coarse-to-fine search strategy and use logarithmic sampling for learning rate and regularization strength
• Design train/dev/test splits appropriate for big-data scenarios (1M+ examples) vs. small-data scenarios
• Diagnose train/dev distribution mismatch and propose corrective strategies (data synthesis, re-weighting)
• Perform structured error analysis by manually inspecting misclassified examples and building error breakdown tables
• Apply orthogonalization: fix high bias first (bigger model, longer training), then fix high variance (more data, regularization)
• Use human-level performance as a proxy for Bayes optimal error to calculate avoidable bias
• Implement a hyperparameter search loop and a fastai-style LR finder from scratch in Python
• Evaluate when to use "Panda" vs. "Caviar" strategy for hyperparameter tuning based on compute budget

Opening Hook

Core Concepts

This chapter covers two tightly related topics: (A) Hyperparameter Tuning — the mechanics of finding good values, and (B) ML Strategy — the decision framework for what to work on next. Together, they form the practitioner's toolkit for going from a working prototype to a production-quality model.

The Hyperparameter Landscape

Not all hyperparameters are created equal. Andrew Ng's practical hierarchy, refined across years of production ML projects, ranks them into four tiers of importance:

Grid Search vs. Random Search

Grid Search: The Naïve Approach

In grid search, you define a set of values for each hyperparameter and try every combination:

Python# Grid search: 5 values × 5 values = 25 experiments
learning_rates = [0.001, 0.003, 0.01, 0.03, 0.1]
hidden_units   = [64, 128, 256, 512, 1024]

for lr in learning_rates:
    for hu in hidden_units:
        train_and_evaluate(lr=lr, hidden=hu)  # 25 runs

Problem: If learning rate matters much more than hidden units (which it does — Tier 1 vs. Tier 2), then in a 5×5 grid you only test 5 unique learning rates. The other 20 experiments are "wasted" exploring hidden unit values when α is already suboptimal.

Random Search: The Better Alternative

In random search, you sample each hyperparameter independently from a range:

Pythonimport numpy as np

# Random search: 25 experiments, but 25 UNIQUE learning rates!
for trial in range(25):
    lr = 10 ** np.random.uniform(-4, -1)   # log-uniform: 0.0001 to 0.1
    hu = np.random.choice([64, 128, 256, 512, 1024])
    train_and_evaluate(lr=lr, hidden=hu)

Advantage: Now you test 25 unique learning rates instead of just 5. Since α matters most, random search explores the most important dimension much more richly.

Coarse-to-Fine Strategy & Logarithmic Scale

The Coarse-to-Fine Workflow

In practice, you don't run one massive search. You iterate in rounds:

1. Round 1 (Coarse): Sample broadly. LR from 10⁻⁴ to 10⁰, hidden units from 32 to 2048. Run ~25 experiments with fewer epochs (5–10).
2. Identify the promising region: e.g., best results cluster around LR ∈ [10⁻³, 10⁻²] and hidden units ∈ [256, 512].
3. Round 2 (Fine): Zoom into the promising region. LR from 10⁻³ to 10⁻², hidden units from 200 to 600. Run ~25 more experiments with more epochs (20–50).
4. Round 3 (Final): Narrow further if needed. Train the top 3 candidates with full epochs and pick the best.

Why Logarithmic Scale for Learning Rate?

Learning rate values that "matter" are spread across orders of magnitude. The difference between 0.0001 and 0.001 is just as significant as the difference between 0.01 and 0.1 — each is a 10× change. If you sampled uniformly from [0.0001, 1], you'd spend 90% of your samples in [0.1, 1] and only 0.1% of samples in [0.0001, 0.001].

Pythonimport numpy as np

# ✅ CORRECT: Log-uniform sampling for learning rate
r = np.random.uniform(-4, -1)  # exponent between -4 and -1
alpha = 10 ** r                   # α ∈ [0.0001, 0.1]

# ❌ WRONG: Uniform sampling for learning rate
alpha = np.random.uniform(0.0001, 0.1)  # heavily biased toward large values

# ✅ Log-uniform for β (momentum): sample (1 - β) on log scale
# β ∈ [0.9, 0.999] → (1-β) ∈ [0.001, 0.1] → r ∈ [-3, -1]
r = np.random.uniform(-3, -1)
beta = 1 - 10 ** r               # β ∈ [0.9, 0.999]

Panda 🐼 vs. Caviar 🐟 Strategy

Train/Dev/Test Split Strategy

The Classical Split

Traditionally (pre-deep learning era, small datasets of 100–10,000 examples):

The Big Data Split

In the deep learning era with 1M+ examples, you don't need 200K examples just for dev set. Modern splits:

Even 0.25% of 10M = 25,000 examples — more than enough to get a statistically significant performance estimate.

Train/Dev Distribution Mismatch

The Real-World Problem

In many production ML systems, your training data comes from a different distribution than what you'll see at inference time:

Why Does Mismatch Happen?

• Data availability: You have millions of web-scraped images but only 10,000 images from your actual camera app
• Temporal shift: Training on historical data, deploying on current data
• Geographic shift: Training on US data, deploying for Indian users
• Platform shift: Training on desktop clicks, deploying on mobile

The Solution: Prioritize Target Distribution for Dev/Test

Always make your dev and test sets reflect the target distribution (what you'll see in production). Use the mismatched (but larger) data for training.

Error Analysis

The Power of Manual Inspection

Before spending weeks collecting more data or redesigning your architecture, spend 30–60 minutes manually inspecting 100 misclassified dev-set examples. This simple practice is one of the highest-ROI activities in ML.

Structured Error Analysis Process

1. Pull 100 misclassified examples from the dev set
2. Create a spreadsheet with columns for each potential error category
3. For each example, mark which categories apply
4. Count percentages to identify the biggest error sources
5. Prioritize the category with the highest ceiling for improvement

Example: Food Classification for Zomato

Imagine you're building a food image classifier for Zomato's photo-based search. You inspect 100 misclassified images:

Conclusion: Working on handling blurry images (data augmentation, super-resolution preprocessing) could fix up to 38% of errors — that's the highest-impact area. Don't waste time on rare regional dishes (only 7% ceiling).

Should You Fix Incorrect Labels?

Deep learning algorithms are robust to random label noise in the training set — if errors are random (not systematic), a small percentage (1–2%) of wrong labels usually doesn't hurt much. However:

• Dev/test set labels must be correct. If 6% of your dev set is mislabeled, you can't trust your model selection process.
• Systematic errors are dangerous. If all "dosa" images are labeled "uttapam", the model will learn this wrong mapping.
• If you fix dev labels, fix test labels too — they must stay from the same distribution.

Orthogonalization

One Knob, One Function

In a well-designed system, each control affects exactly one thing. In an old TV: brightness knob changes brightness, volume knob changes volume. If one knob changed both, debugging would be impossible.

Apply the same principle to ML. The four sequential goals, each with its own "knob":

Human-Level Performance & Bayes Optimal Error

Why Compare to Humans?

For tasks that humans are good at (vision, speech, NLP), human-level error is a useful proxy for the Bayes optimal error — the theoretical best any function can achieve given the noise in the data.

Which "Human Level" to Use?

Consider a medical imaging task:

For the purpose of computing avoidable bias, use the best human performance (0.7%) as the proxy for Bayes error, because Bayes error ≤ 0.7%.

Worked Diagnostic Examples

Scenario A: High Avoidable Bias

Avoidable bias = 8% − 1% = 7%
Variance = 10% − 8% = 2%
Diagnosis: Focus on reducing bias → bigger model, train longer.

Scenario B: High Variance

Avoidable bias = 2% − 1% = 1%
Variance = 10% − 2% = 8%
Diagnosis: Focus on reducing variance → regularization, more data, dropout.

Surpassing Human-Level Performance

Once a model surpasses human-level error, progress typically slows down because:

• You can no longer use human-level as a reliable Bayes proxy — the gap becomes unclear
• You can't do manual error analysis (if the model is better than you, how do you know which errors to fix?)
• You're approaching the theoretical ceiling (Bayes error), where further gains require exponentially more effort

From-Scratch Code

4.1 Hyperparameter Random Search Loop

A complete, reusable hyperparameter search engine from scratch:

Pythonimport numpy as np
import json
from datetime import datetime

class HyperparameterSearcher:
    """Random search over hyperparameters with logging."""

    def __init__(self, search_space, n_trials=25, seed=42):
        """
        search_space: dict mapping param_name -> dict with:
            'type': 'log_uniform' | 'uniform' | 'choice' | 'int_uniform'
            'low', 'high': for uniform/log_uniform/int_uniform
            'options': list for choice
        """
        self.search_space = search_space
        self.n_trials = n_trials
        self.rng = np.random.RandomState(seed)
        self.results = []

    def _sample_params(self):
        """Sample one set of hyperparameters."""
        params = {}
        for name, spec in self.search_space.items():
            if spec['type'] == 'log_uniform':
                # Sample on log scale: 10^Uniform(low, high)
                r = self.rng.uniform(spec['low'], spec['high'])
                params[name] = 10 ** r
            elif spec['type'] == 'uniform':
                params[name] = self.rng.uniform(spec['low'], spec['high'])
            elif spec['type'] == 'choice':
                params[name] = self.rng.choice(spec['options'])
            elif spec['type'] == 'int_uniform':
                params[name] = int(self.rng.uniform(spec['low'], spec['high']))
            elif spec['type'] == 'log_complement':
                # For β: sample (1-β) on log scale
                r = self.rng.uniform(spec['low'], spec['high'])
                params[name] = 1 - 10 ** r
        return params

    def search(self, train_fn):
        """
        Run the search.
        train_fn: callable(params_dict) -> dict with 'train_loss',
                  'dev_loss', 'dev_acc', etc.
        """
        print(f"Starting random search: {self.n_trials} trials")
        print("=" * 60)

        for trial in range(self.n_trials):
            params = self._sample_params()
            print(f"\nTrial {trial+1}/{self.n_trials}")
            print(f"  Params: {params}")

            # Train and evaluate
            metrics = train_fn(params)

            # Log results
            result = {
                'trial': trial + 1,
                'params': params,
                'metrics': metrics,
                'timestamp': datetime.now().isoformat()
            }
            self.results.append(result)

            print(f"  Dev Acc: {metrics.get('dev_acc', 'N/A')}")

        # Find best trial
        best = max(self.results,
                   key=lambda x: x['metrics'].get('dev_acc', 0))
        print(f"\n{'='*60}")
        print(f"Best Trial: #{best['trial']}")
        print(f"Best Params: {best['params']}")
        print(f"Best Dev Acc: {best['metrics']['dev_acc']:.4f}")
        return best

    def top_k(self, k=5):
        """Return top-k trials by dev accuracy."""
        sorted_results = sorted(
            self.results,
            key=lambda x: x['metrics'].get('dev_acc', 0),
            reverse=True
        )
        return sorted_results[:k]


# ─── Usage Example ───
search_space = {
    'learning_rate': {'type': 'log_uniform', 'low': -4, 'high': -1},
    'hidden_units':  {'type': 'choice', 'options': [64,128,256,512]},
    'dropout_rate':  {'type': 'uniform', 'low': 0.1, 'high': 0.5},
    'momentum':      {'type': 'log_complement', 'low': -3, 'high': -1},
    'batch_size':    {'type': 'choice', 'options': [32,64,128,256]},
}

searcher = HyperparameterSearcher(search_space, n_trials=25)
# best = searcher.search(my_train_function)

4.2 Learning Rate Finder (fastai-style)

The LR Finder is one of the most practical tools in deep learning. It trains for one epoch while exponentially increasing the learning rate, and records the loss at each step. The optimal LR is where the loss decreases fastest (steepest slope).

Pythonimport numpy as np
import copy

class LRFinder:
    """
    Learning Rate Finder (Smith 2017 / fastai-style).
    Exponentially increases LR from lr_min to lr_max over
    one pass through the data, recording loss at each step.
    """

    def __init__(self, model, optimizer_fn, loss_fn):
        """
        model:        object with .forward(X) and .parameters()
        optimizer_fn: callable(params, lr) -> optimizer
        loss_fn:      callable(y_pred, y_true) -> scalar loss
        """
        self.model = model
        self.optimizer_fn = optimizer_fn
        self.loss_fn = loss_fn
        self.lrs = []
        self.losses = []

    def find(self, X_train, y_train, lr_min=1e-7, lr_max=10,
             num_steps=100, smooth_factor=0.05):
        """Run the LR range test."""

        # Save initial model state
        initial_state = copy.deepcopy(self.model)

        # Compute multiplication factor per step
        mult = (lr_max / lr_min) ** (1 / num_steps)
        lr = lr_min
        best_loss = float('inf')
        avg_loss = 0
        n = len(X_train)
        batch_size = min(64, n)

        for step in range(num_steps):
            # Sample a mini-batch
            idx = np.random.choice(n, batch_size, replace=False)
            X_batch = X_train[idx]
            y_batch = y_train[idx]

            # Forward pass
            y_pred = self.model.forward(X_batch)
            loss = self.loss_fn(y_pred, y_batch)

            # Smooth the loss (exponential moving average)
            avg_loss = smooth_factor * loss + (1 - smooth_factor) * avg_loss
            smoothed_loss = avg_loss / (1 - (1 - smooth_factor) ** (step + 1))

            # Stop if loss explodes (> 4× best)
            if step > 10 and smoothed_loss > 4 * best_loss:
                print(f"Stopping: loss exploded at lr={lr:.2e}")
                break

            if smoothed_loss < best_loss:
                best_loss = smoothed_loss

            # Record
            self.lrs.append(lr)
            self.losses.append(smoothed_loss)

            # Backward pass & update (simplified)
            optimizer = self.optimizer_fn(
                self.model.parameters(), lr=lr
            )
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            # Increase learning rate exponentially
            lr *= mult

        # Restore initial model state
        self.model = initial_state
        return self.lrs, self.losses

    def suggest_lr(self):
        """Suggest the LR where loss decreases fastest."""
        if len(self.losses) < 3:
            return None

        # Compute gradient of loss w.r.t. log(lr)
        log_lrs = np.log10(self.lrs)
        gradients = np.gradient(self.losses, log_lrs)

        # Find the LR with steepest negative gradient
        # (skip first 10% and last 10% for stability)
        start = len(gradients) // 10
        end = len(gradients) - len(gradients) // 10
        min_idx = start + np.argmin(gradients[start:end])

        suggested_lr = self.lrs[min_idx]
        print(f"Suggested LR: {suggested_lr:.2e}")
        print(f"  (where loss decreased fastest)")
        print(f"  Rule of thumb: use ~{suggested_lr/10:.2e} to "
              f"{suggested_lr:.2e}")
        return suggested_lr

4.3 Error Analysis Helper

Pythonimport numpy as np
from collections import Counter

class ErrorAnalyzer:
    """Structured error analysis on misclassified examples."""

    def __init__(self, X_dev, y_dev, y_pred, class_names=None):
        self.X_dev = X_dev
        self.y_dev = y_dev
        self.y_pred = y_pred
        self.class_names = class_names

        # Find misclassified indices
        self.misclassified = np.where(y_dev != y_pred)[0]
        self.total_errors = len(self.misclassified)
        self.dev_error = self.total_errors / len(y_dev)

        print(f"Dev set size: {len(y_dev)}")
        print(f"Misclassified: {self.total_errors}")
        print(f"Dev error rate: {self.dev_error:.2%}")

    def confusion_breakdown(self):
        """Show which classes are most confused."""
        confusion_pairs = []
        for idx in self.misclassified:
            true_label = self.y_dev[idx]
            pred_label = self.y_pred[idx]
            if self.class_names:
                pair = (self.class_names[true_label],
                        self.class_names[pred_label])
            else:
                pair = (true_label, pred_label)
            confusion_pairs.append(pair)

        counts = Counter(confusion_pairs)
        print("\nTop Confusion Pairs (True → Predicted):")
        print("-" * 50)
        for (true, pred), count in counts.most_common(10):
            pct = count / self.total_errors * 100
            print(f"  {true:>15s} → {pred:<15s}  "
                  f"{count:4d}  ({pct:.1f}%)")
        return counts

    def ceiling_analysis(self, categories):
        """
        Given error categories with counts, compute ceilings.
        categories: dict mapping category_name -> count_of_errors
        """
        print(f"\nError Ceiling Analysis (Dev Error: {self.dev_error:.2%})")
        print("-" * 55)
        print(f"{'Category':<25s} {'Count':>6s} {'% Errors':>9s} {'Ceiling':>9s}")
        print("-" * 55)
        for cat, count in sorted(categories.items(),
                                  key=lambda x: -x[1]):
            pct = count / self.total_errors * 100
            ceiling = self.dev_error * (1 - count / self.total_errors)
            print(f"  {cat:<23s} {count:6d} {pct:8.1f}% "
                  f"{ceiling:8.2%}")
        print("-" * 55)

Industry Code — PyTorch & Optuna

5.1 PyTorch LR Finder (torch-lr-finder)

Pythonimport torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
from torch_lr_finder import LRFinder  # pip install torch-lr-finder

# Define a simple model
model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.BatchNorm1d(256),
    nn.Dropout(0.3),
    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Linear(128, 10)
)

optimizer = optim.Adam(model.parameters(), lr=1e-7, weight_decay=1e-4)
criterion = nn.CrossEntropyLoss()

# Create data loader
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)

# Run LR Finder
lr_finder = LRFinder(model, optimizer, criterion, device="cuda")
lr_finder.range_test(train_loader, end_lr=10, num_iter=200)
lr_finder.plot()  # Shows loss vs. LR curve
lr_finder.reset()  # Restore model to initial state

# Read the plot: pick LR where loss is steepest
# Typically: suggested_lr ≈ 3e-3 for this architecture

5.2 Optuna — Automated Hyperparameter Search

Pythonimport optuna
import torch
import torch.nn as nn
import torch.optim as optim

def objective(trial):
    """Optuna objective function for HP search."""

    # ── Sample hyperparameters ──
    lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True)
    n_layers = trial.suggest_int("n_layers", 2, 5)
    dropout = trial.suggest_float("dropout", 0.1, 0.5)
    hidden_size = trial.suggest_categorical(
        "hidden_size", [64, 128, 256, 512]
    )
    batch_size = trial.suggest_categorical(
        "batch_size", [32, 64, 128]
    )

    # ── Build model dynamically ──
    layers = []
    in_dim = 784
    for i in range(n_layers):
        layers.append(nn.Linear(in_dim, hidden_size))
        layers.append(nn.ReLU())
        layers.append(nn.Dropout(dropout))
        in_dim = hidden_size
    layers.append(nn.Linear(in_dim, 10))
    model = nn.Sequential(*layers)

    optimizer = optim.Adam(model.parameters(), lr=lr)
    criterion = nn.CrossEntropyLoss()

    # ── Training loop (simplified) ──
    for epoch in range(20):
        model.train()
        for X_batch, y_batch in train_loader:
            optimizer.zero_grad()
            loss = criterion(model(X_batch), y_batch)
            loss.backward()
            optimizer.step()

        # ── Evaluate on dev set ──
        model.eval()
        correct = 0
        total = 0
        with torch.no_grad():
            for X_val, y_val in dev_loader:
                preds = model(X_val).argmax(dim=1)
                correct += (preds == y_val).sum().item()
                total += len(y_val)
        dev_acc = correct / total

        # Pruning: stop bad trials early
        trial.report(dev_acc, epoch)
        if trial.should_prune():
            raise optuna.exceptions.TrialPruned()

    return dev_acc

# ── Run the study ──
study = optuna.create_study(
    direction="maximize",
    pruner=optuna.pruners.MedianPruner(n_warmup_steps=5)
)
study.optimize(objective, n_trials=50, timeout=3600)  # 1-hour budget

# ── Results ──
print(f"Best trial: {study.best_trial.number}")
print(f"Best accuracy: {study.best_value:.4f}")
print(f"Best params: {study.best_params}")

# Visualize (optional)
optuna.visualization.plot_optimization_history(study)
optuna.visualization.plot_param_importances(study)

Visual Diagrams

6.1 Grid Search vs. Random Search — Visual

6.2 Error Decomposition Waterfall

6.3 The LR Finder Plot

6.4 Orthogonalization — The Four Knobs

Worked Example — Complete HP Tuning Workflow

Problem: MNIST Digit Classifier at ₹500 Cloud Budget

You're a student at VIT Vellore. Your professor gives you a ₹500 Google Cloud credit and asks you to build the best MNIST classifier you can. You have a single T4 GPU (training one model takes ~3 minutes). Budget allows ~100 experiments.

Step 1: Define the Search Space (Using Tier Priorities)

Step 2: Round 1 — Coarse Search (30 trials, 5 epochs each)

30 trials × 5 epochs × 3 min = 15 GPU-minutes. Cost: ~₹10.

Python# Round 1 results (top 5 of 30 trials)
# Trial | LR       | Units | Batch | Drop | Layers | Dev Acc
# ──────┼──────────┼───────┼───────┼──────┼────────┼────────
#   7   | 3.2e-3   |  256  |   64  | 0.30 |   3    | 97.8%
#  12   | 1.8e-3   |  512  |   64  | 0.25 |   3    | 97.6%
#  19   | 5.1e-3   |  256  |  128  | 0.35 |   2    | 97.5%
#  23   | 2.4e-3   |  128  |   64  | 0.20 |   3    | 97.3%
#   3   | 8.7e-3   |  256  |   32  | 0.40 |   2    | 97.1%

Observation: Best LRs cluster around [1e-3, 1e-2]. Hidden units 256–512. 3 layers seems best. Batch size 64. Dropout 0.20–0.35.

Step 3: Round 2 — Fine Search (20 trials, 20 epochs each)

Narrow the ranges based on Round 1 findings:

Python# Narrowed search space
fine_search_space = {
    'lr':      {'type': 'log_uniform', 'low': np.log10(1e-3),
                'high': np.log10(1e-2)},   # narrowed!
    'units':   {'type': 'choice', 'options': [192,256,320,384,512]},
    'dropout': {'type': 'uniform', 'low': 0.2, 'high': 0.4},
    'layers':  {'type': 'choice', 'options': [3, 4]},
}

# Top result from Round 2:
# LR = 2.8e-3, units = 320, dropout = 0.28, layers = 3
# Dev Accuracy: 98.4%

Step 4: Final Training (Top 3 candidates, full 50 epochs)

Python# Final evaluation on held-out test set
# Model          | Dev Acc  | Test Acc
# ───────────────┼──────────┼──────────
# Candidate A    | 98.4%    | 98.3%    ← Selected (dev ≈ test, no overfitting)
# Candidate B    | 98.3%    | 98.2%
# Candidate C    | 98.1%    | 98.0%

Step 5: Error Analysis on the Best Model

Inspect 100 misclassified test examples:

Conclusion: To go beyond 98.3%, focus on data augmentation for rotation/skew (28% ceiling) and potentially ensemble models for ambiguous digits. Total cloud spend: ~₹80 out of ₹500 budget. 🎯

Case Study — Ola Surge Pricing

Common Mistakes & Misconceptions

Comparison Tables

10.1 Search Strategies Compared

10.2 Bias vs. Variance vs. Mismatch Diagnostics

10.3 Panda vs. Caviar Strategy

Exercises

Multiple Choice Questions (10)

Short Answer Questions (5)

B1. Beginner

List the four tiers of hyperparameter importance according to Andrew Ng. Give one example hyperparameter for each tier.

B2. Intermediate

Explain with a numerical example why sampling the momentum parameter β uniformly from [0.9, 0.999] is a bad idea. What should you do instead?

B3. Intermediate

You're building a crop disease classifier for an agri-tech startup in Pune. You have 2 million images from web-scraped agricultural databases but only 8,000 images from Indian farmers' phones (the target use case). How would you design the train/dev/test split?

B4. Intermediate

A model has: Human error = 3%, Train error = 4%, Train-Dev error = 5%, Dev error = 12%, Test error = 13%. Compute all four error gaps (avoidable bias, variance, data mismatch, dev overfitting) and identify the top priority to fix.

B5. Advanced

Explain why ML progress typically slows down after surpassing human-level performance. Give two concrete reasons related to the ML workflow.

Long Answer Questions (3)

C1. Intermediate — The Complete Tuning Workflow (10 marks)

You are building a text classification model to categorize customer complaints for a telecom company (Jio). You have 5 million labeled complaints from email (historical) and need the model to work on WhatsApp messages (target). Describe a complete ML strategy covering: (a) data split design, (b) hyperparameter search approach, (c) error analysis methodology, (d) how to handle distribution mismatch.

C2. Advanced — Grid vs. Random: Mathematical Proof (8 marks)

Prove mathematically that for a fixed budget of N experiments, random search tests more unique values along the most important hyperparameter axis than grid search when there are k hyperparameters (k ≥ 2). State your assumptions clearly. Compute the exact number of unique values per axis for both methods when N = 64 and k = 3.

C3. Advanced — Orthogonalization in Practice (8 marks)

A junior ML engineer at Infosys reports the following results for a sentiment analysis model: Human error = 2%, Training error = 15%, Dev error = 16%. They propose: "Let's add dropout and L2 regularization since the dev error is high." Critique this proposal using orthogonalization principles. What would you recommend instead? Explain your reasoning step by step.

Programming Exercises (2)

D1. Intermediate — Build a Complete HP Search Pipeline

Using the HyperparameterSearcher class from Section 4, build a complete pipeline that:

1. Defines a search space for a 3-layer neural network (LR, hidden units, dropout, batch size)
2. Implements a dummy train_fn that simulates training (you can use synthetic data or sklearn's make_classification)
3. Runs a 2-round coarse-to-fine search (Round 1: 20 trials, broad range; Round 2: 10 trials, narrowed range)
4. Prints the top-5 results from each round
5. Reports the final best hyperparameters

Deliverable: A Python script that runs end-to-end and prints results to console.

D2. Advanced — LR Finder with PyTorch

Implement a LR Finder for a PyTorch model on the Fashion-MNIST dataset:

1. Load Fashion-MNIST using torchvision.datasets
2. Define a 3-layer network with BatchNorm and Dropout
3. Implement the LR range test: exponentially increase LR from 10⁻⁷ to 10 over one epoch
4. Plot the loss vs. log(LR) curve using matplotlib
5. Automatically find and print the suggested LR (steepest descent point)
6. Train the model using the suggested LR and report test accuracy

Deliverable: A Python script with a matplotlib plot and final test accuracy ≥ 88%.

Mini-Project

🏗️ End-to-End ML Strategy for Indian Language Sentiment Analysis

Scenario: You're building a sentiment classifier for Hinglish (Hindi-English code-mixed) product reviews for a Flipkart internship project. You have:

• 500K English Amazon reviews (labeled positive/negative) — source distribution
• 5K Hinglish Flipkart reviews (labeled) — target distribution
• 50K unlabeled Hinglish reviews

Tasks:

1. Data Split Design: Design your train/dev/test/train-dev splits. Justify your choices with exact numbers.
2. Baseline Model: Train a simple logistic regression or 2-layer NN on English data. Report train, train-dev, dev, and test errors.
3. Error Decomposition: Compute avoidable bias, variance, and data mismatch. Identify the biggest problem.
4. Hyperparameter Search: Run a 2-round coarse-to-fine random search using Optuna. Document all trials.
5. Error Analysis: Manually inspect 50 misclassified dev examples. Create an error category spreadsheet and perform ceiling analysis.
6. Strategy Proposal: Based on your error analysis, propose 3 concrete next steps with expected impact. Prioritize using ceilings.

Deliverable: A Jupyter notebook with all code, analysis tables, and a 500-word strategy writeup. Grading emphasizes strategic reasoning over raw accuracy.

Chapter Summary

References & Further Reading

Foundational Papers

1. Bergstra, J., & Bengio, Y. (2012). Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research, 13, 281–305. — The seminal paper proving random search superiority over grid search.
2. Smith, L.N. (2017). Cyclical Learning Rates for Training Neural Networks. IEEE WACV. — Introduced the LR range test (LR Finder) and cyclical learning rates.
3. Snoek, J., Larochelle, H., & Adams, R.P. (2012). Practical Bayesian Optimization of Machine Learning Algorithms. NeurIPS. — Foundations of Bayesian hyperparameter optimization.
4. Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A Next-generation Hyperparameter Optimization Framework. KDD. — The Optuna paper.

Textbooks & Courses

5. Ng, A. (2017). Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization. Coursera Deep Learning Specialization, Course 2. — Primary source for the tier ranking, orthogonalization, and error analysis frameworks.
6. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press, Chapter 11: Practical Methodology. — Covers hyperparameter selection strategy.
7. Howard, J., & Gugger, S. (2020). Deep Learning for Coders with fastai and PyTorch. O'Reilly. — Practical LR Finder and 1cycle policy implementation.

Indian Industry Applications

8. Ola Engineering Blog (2023). Dynamic Pricing at Scale: ML Architecture Behind Surge Pricing. engineering.olacabs.com
9. Flipkart Tech Blog (2023). Scaling Recommendations for 400M Users. tech.flipkart.com
10. Niramai Health Analytix. AI for Breast Cancer Screening. niramai.com — Affordable AI-powered diagnostics using thermal imaging.

Tools & Libraries

11. Optuna Documentation: optuna.readthedocs.io
12. torch-lr-finder: github.com/davidtvs/pytorch-lr-finder
13. Weights & Biases Sweeps: wandb.ai/site/sweeps
14. Ray Tune: docs.ray.io/en/latest/tune/