Chapter 17: Transfer Learning and Fine-Tuning

Bloom's Taxonomy Map for This Chapter

Learning Objectives

By the end of this chapter, you will be able to:

• Distinguish feature extraction (frozen backbone + new head) from fine-tuning (unfrozen backbone + end-to-end training) and explain when each strategy is optimal
• Implement freeze strategies: full freeze, partial freeze, gradual unfreezing, and discriminative learning rates using PyTorch's parameter groups
• Derive why lower layers of a pretrained CNN capture universal edge/texture detectors that transfer across domains, using the feature hierarchy framework
• Apply HuggingFace's Trainer API to fine-tune BERT for Hindi sentiment analysis with under 50 lines of code
• Explain LoRA's low-rank decomposition W + ΔW = W + BA, where B∈ℝ^d×r and A∈ℝ^r×k, and compute the parameter savings for a given rank r
• Diagnose negative transfer, understand domain shift, and select appropriate strategies based on dataset size and domain similarity
• Design a few-shot and zero-shot classification system using foundation models and prompt engineering

Opening Hook

The Intuition First

The "Learning to Drive" Analogy

Imagine you learned to drive a car in Mumbai — fighting through auto-rickshaws, potholes, and traffic that follows no known laws of physics. Now you move to San Francisco. Do you re-learn driving from scratch? Of course not!

Your low-level skills transfer perfectly: steering, braking, reading road signs, judging distances. These are like the early layers of a neural network — they detect universal patterns (edges, textures, colors) that work everywhere.

Your mid-level skills mostly transfer: lane discipline, highway merging, parking. But you'll need to adjust — Americans drive on the right. These are like the middle layers of a CNN — spatial patterns that are similar but need calibration.

Your high-level skills need the most adaptation: knowing which roads are one-way, what a "California stop" is, how four-way stops work. These are like the final layers of a network — task-specific knowledge that changes between domains.

"Aha!" Question

🤔 Think about this: A ResNet50 trained on ImageNet (14 million images of 1000 categories — dogs, cars, planes) has never seen an Indian saree, a plate of dosa, or a bottle of Limca. Yet when Flipkart uses this ResNet50 to classify Indian products, it achieves 92% accuracy with just 5,000 training images. How is this possible?

The answer: The first 48 layers of ResNet50 have learned to detect edges, textures, corners, color gradients, fabric patterns, circular shapes, and complex object parts. A saree is ultimately made of fabric textures (which ImageNet learned from curtains and carpets), color patterns (learned from flowers and paintings), and draped shapes (learned from clothing categories). Only the final classification layer needs to learn "this combination of features = saree."

Why Transfer Learning Works

The Mathematical Foundation

Let's build the theory from first principles. Consider two tasks:

• Source task T_S: ImageNet classification (1.2M images, 1000 classes)
• Target task T_T: Flipkart product categorization (5,000 images, 50 classes)

A deep neural network f(x; θ) can be decomposed as:

The key insight is that φ learns a feature representation that maps raw pixels to a semantically meaningful vector space. Transfer learning works when the feature space learned for T_S is also useful for T_T.

The Feature Hierarchy

Zeiler and Fergus (2014) visualized what each layer of a CNN learns. Here's what you see:

The Four Quadrants Decision Framework

When should you use which strategy? It depends on two factors: dataset size and domain similarity.

Feature Extraction vs. Fine-Tuning

Strategy 1: Feature Extraction (Frozen Backbone)

In feature extraction, you treat the pretrained model as a fixed feature calculator. You freeze all backbone parameters and only train a new classification head.

Strategy 2: Fine-Tuning (Unfrozen Backbone)

In fine-tuning, you unfreeze some or all of the backbone layers and train them along with the new head, typically with a smaller learning rate.

Side-by-Side Comparison

Freeze Strategies: What to Freeze and When

Strategy 1: Full Freeze (Feature Extraction)

Freeze the entire backbone. Only the classifier head trains.

PyTorch
# Full freeze — only train the final classifier
import torchvision.models as models

model = models.resnet50(pretrained=True)

# Freeze ALL backbone parameters
for param in model.parameters():
    param.requires_grad = False

# Replace final FC layer (unfrozen by default)
model.fc = nn.Linear(2048, num_classes)  # Only this trains

# Verify: count trainable parameters
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"Training {trainable:,} / {total:,} params ({100*trainable/total:.2f}%)")
# Output: Training 102,450 / 25,659,954 params (0.40%)

Strategy 2: Partial Freeze

Freeze early layers (universal features), unfreeze later layers (task-specific).

PyTorch
# Partial freeze — freeze layers 1-3, unfreeze layer4 + fc
model = models.resnet50(pretrained=True)

# Freeze everything first
for param in model.parameters():
    param.requires_grad = False

# Unfreeze layer4 (the last residual block)
for param in model.layer4.parameters():
    param.requires_grad = True

# Replace and unfreeze classifier
model.fc = nn.Linear(2048, num_classes)

# Now layer4 + fc are trainable, everything else frozen
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Training {trainable:,} params")
# Output: Training 7,176,242 params (27.97%)

Strategy 3: Gradual Unfreezing (Howard & Ruder, ULMFiT 2018)

This is the most sophisticated strategy. You start with everything frozen, then progressively unfreeze layers from top to bottom over training epochs.

PyTorch
# Gradual unfreezing implementation
class GradualUnfreezer:
    def __init__(self, model, layer_groups):
        """
        layer_groups: list of nn.Module groups, ordered bottom→top
        Example for ResNet50: [model.layer1, model.layer2, 
                               model.layer3, model.layer4, model.fc]
        """
        self.model = model
        self.layer_groups = layer_groups
        self.current_group = len(layer_groups) - 1  # Start with last
        
        # Freeze everything
        for param in model.parameters():
            param.requires_grad = False
        
        # Unfreeze only the last group (classifier)
        for param in layer_groups[-1].parameters():
            param.requires_grad = True
    
    def unfreeze_next(self):
        """Call this at the start of each epoch"""
        if self.current_group > 0:
            self.current_group -= 1
            for param in self.layer_groups[self.current_group].parameters():
                param.requires_grad = True
            print(f"Unfroze group {self.current_group}: "
                  f"{self.layer_groups[self.current_group].__class__.__name__}")

# Usage
model = models.resnet50(pretrained=True)
model.fc = nn.Linear(2048, 50)
unfreezer = GradualUnfreezer(model, [
    model.layer1, model.layer2, model.layer3, model.layer4, model.fc
])

for epoch in range(10):
    if epoch > 0 and epoch % 2 == 0:
        unfreezer.unfreeze_next()  # Unfreeze one more group
    train_one_epoch(model, train_loader, optimizer)

Discriminative Learning Rates & Gradual Unfreezing

The Core Idea

Different layers need different learning rates. Early layers already have excellent features — you want to nudge them gently. Later layers need bigger updates. The classifier head, starting from random weights, needs the largest learning rate.

PyTorch Implementation with Parameter Groups

PyTorch
import torch.optim as optim

model = models.resnet50(pretrained=True)
model.fc = nn.Linear(2048, 50)  # 50 product categories

# Define parameter groups with discriminative learning rates
base_lr = 3e-3
factor = 2.6

param_groups = [
    {'params': model.conv1.parameters(),   'lr': base_lr / factor**4},  # 6.56e-5
    {'params': model.layer1.parameters(),  'lr': base_lr / factor**3},  # 1.71e-4
    {'params': model.layer2.parameters(),  'lr': base_lr / factor**2},  # 4.44e-4
    {'params': model.layer3.parameters(),  'lr': base_lr / factor**1},  # 1.15e-3
    {'params': model.layer4.parameters(),  'lr': base_lr},             # 3.00e-3
    {'params': model.fc.parameters(),      'lr': base_lr * 3},          # 9.00e-3
]

optimizer = optim.Adam(param_groups, weight_decay=1e-4)

# Print to verify
for i, pg in enumerate(optimizer.param_groups):
    n_params = sum(p.numel() for p in pg['params'])
    print(f"Group {i}: lr={pg['lr']:.6f}, params={n_params:,}")

Domain Adaptation

The Problem: Domain Shift

Transfer learning assumes source and target domains are "close enough." But what happens when they're not? This is domain shift — the statistical difference between training data and deployment data.

Domain Adaptation Techniques

1. Feature-Level Adaptation (DANN — Ganin et al., 2016)

Add a domain discriminator that tries to distinguish source features from target features, and train the feature extractor to fool it (adversarial approach).

2. Instance-Level Adaptation (Importance Weighting)

Reweight source samples to match target distribution:

3. Self-Training (Pseudo-Labels)

Use the source-trained model to generate pseudo-labels on target data, then retrain:

1. Train model on labeled source data
2. Predict labels for unlabeled target data
3. Keep high-confidence predictions as pseudo-labels
4. Retrain on source labels + pseudo-labels
5. Repeat until convergence

Negative Transfer: When Knowledge Hurts

What Is Negative Transfer?

Negative transfer occurs when transferring knowledge from the source task decreases performance on the target task — the pretrained model is worse than training from scratch.

How to Detect Negative Transfer

Python
# Negative transfer detection protocol
def detect_negative_transfer(pretrained_model, scratch_model, 
                              train_data, val_data, n_epochs=20):
    """Compare pretrained vs from-scratch performance"""
    
    # Track validation accuracy over epochs
    pretrained_accs = []
    scratch_accs = []
    
    for epoch in range(n_epochs):
        # Train both models
        train_one_epoch(pretrained_model, train_data)
        train_one_epoch(scratch_model, train_data)
        
        # Evaluate
        pt_acc = evaluate(pretrained_model, val_data)
        sc_acc = evaluate(scratch_model, val_data)
        pretrained_accs.append(pt_acc)
        scratch_accs.append(sc_acc)
    
    # Negative transfer if scratch beats pretrained consistently
    neg_transfer = all(s > p for s, p in 
                       zip(scratch_accs[-5:], pretrained_accs[-5:]))
    
    if neg_transfer:
        print("⚠️ NEGATIVE TRANSFER DETECTED!")
        print(f"Scratch: {scratch_accs[-1]:.3f} > Pretrained: {pretrained_accs[-1]:.3f}")
        print("Recommendations:")
        print("  1. Try freezing more layers")
        print("  2. Use a different pretrained model")
        print("  3. Reduce learning rate significantly")
        print("  4. Try feature extraction instead of fine-tuning")
    
    return neg_transfer, pretrained_accs, scratch_accs

Mitigation Strategies

Few-Shot and Zero-Shot Learning

The Extreme Case of Transfer Learning

What if you have almost NO labeled data for your target task? This is where few-shot and zero-shot learning — the most aggressive forms of transfer learning — come in.

Zero-Shot Classification with CLIP

Python
# Zero-shot Indian product classification with CLIP
from transformers import CLIPProcessor, CLIPModel
from PIL import Image

# Load pretrained CLIP (never trained on Flipkart products!)
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

# Define categories — just text descriptions, no training!
categories = [
    "a photo of a saree",
    "a photo of a kurta",
    "a photo of jeans",
    "a photo of a cup of chai",
    "a photo of a smartphone",
    "a photo of kolhapuri chappals",
    "a photo of a cricket bat"
]

# Classify an image — zero training data required!
image = Image.open("test_product.jpg")
inputs = processor(text=categories, images=image, 
                   return_tensors="pt", padding=True)
outputs = model(**inputs)

# CLIP computes cosine similarity between image and each text
probs = outputs.logits_per_image.softmax(dim=1)
for cat, prob in zip(categories, probs[0]):
    print(f"{cat}: {prob:.3f}")

Few-Shot with In-Context Learning (GPT-style)

Python
# Few-shot sentiment analysis for Hindi reviews
prompt = """Classify the Hindi review as Positive or Negative.

Review: "यह फोन बहुत अच्छा है, कैमरा शानदार है" → Positive
Review: "बैटरी बहुत जल्दी खत्म हो जाती है" → Negative  
Review: "पैसा वसूल प्रोडक्ट, बहुत खुश हूं" → Positive

Review: "डिलीवरी में बहुत देर लगी और प्रोडक्ट टूटा हुआ आया" →"""

# The model sees 3 examples and generalizes to the 4th
# No fine-tuning required! Just pattern matching in context.

LoRA and Parameter-Efficient Fine-Tuning (PEFT)

The Problem: Fine-Tuning Is Expensive

Fine-tuning a 7B-parameter LLaMA model requires:

• Storing 7B float32 parameters: 28 GB
• Adam optimizer states (2 copies): 56 GB
• Gradients: 28 GB
• Activations for batch size 1: ~20 GB
• Total: ~132 GB — you need 2× A100 80GB GPUs just to fine-tune!

PEFT methods solve this by training only a tiny fraction of parameters while keeping the rest frozen.

LoRA: Low-Rank Adaptation (Hu et al., 2021)

LoRA is the most important PEFT technique. The core insight is beautiful:

LoRA From Scratch in NumPy

NumPy
import numpy as np

class LoRALayer:
    """LoRA adapter from scratch — no frameworks needed"""
    
    def __init__(self, pretrained_W, rank=8, alpha=16):
        """
        pretrained_W: frozen weight matrix (d × k)
        rank: LoRA rank r (typically 4, 8, or 16)
        alpha: scaling factor (typically 2 × rank)
        """
        self.W = pretrained_W  # FROZEN — never updated
        self.d, self.k = pretrained_W.shape
        self.rank = rank
        self.scale = alpha / rank  # Scaling factor
        
        # Initialize LoRA matrices
        # A: random Gaussian initialization
        self.A = np.random.randn(rank, self.k) * 0.01  # r × k
        # B: zero initialization (so ΔW = 0 at start)
        self.B = np.zeros((self.d, rank))               # d × r
        
        # For Adam optimizer
        self.grad_A = None
        self.grad_B = None
    
    def forward(self, x):
        """
        x: input (batch_size × k)
        returns: output (batch_size × d)
        """
        # Frozen pretrained path
        h_pretrained = x @ self.W.T                  # (B, k) × (k, d) = (B, d)
        
        # LoRA path: x → A → B (low-rank update)
        h_lora = x @ self.A.T @ self.B.T             # (B, k)×(k, r)×(r, d) = (B, d)
        h_lora = h_lora * self.scale                 # Scale by α/r
        
        # Combined output
        self.x_cache = x  # Save for backward
        return h_pretrained + h_lora  # h = Wx + (α/r)BAx
    
    def backward(self, grad_output):
        """Compute gradients for A and B only (W is frozen!)"""
        # grad_output: (batch_size × d)
        
        # Gradient for B: dL/dB = (dL/dh)ᵀ · (Ax)ᵀ × scale
        Ax = self.x_cache @ self.A.T  # (B, r)
        self.grad_B = (grad_output.T @ Ax) * self.scale  # (d, r)
        
        # Gradient for A: dL/dA = Bᵀ(dL/dh)ᵀ · x × scale
        self.grad_A = (self.B.T @ grad_output.T @ self.x_cache) * self.scale  # (r, k)
        
        # No gradient for W (frozen!)
        return grad_output @ self.W  # Pass gradient through for chain rule
    
    def update(self, lr=1e-4):
        """Simple SGD update — only A and B change"""
        self.A -= lr * self.grad_A
        self.B -= lr * self.grad_B
    
    def get_merged_weight(self):
        """Merge LoRA into W for inference (no extra cost!)"""
        return self.W + self.scale * (self.B @ self.A)  # d × k
    
    def count_params(self):
        total = self.W.size  # d × k
        trainable = self.A.size + self.B.size  # r(d+k)
        print(f"Total: {total:,} | Trainable: {trainable:,} | "
              f"Ratio: {trainable/total:.4%}")

# Demo
W_pretrained = np.random.randn(4096, 4096)  # Simulated pretrained weight
lora = LoRALayer(W_pretrained, rank=8, alpha=16)
lora.count_params()

x = np.random.randn(4, 4096)  # Batch of 4
output = lora.forward(x)
print(f"Output shape: {output.shape}")

Other PEFT Methods

The HuggingFace Ecosystem

The "GitHub of Machine Learning"

HuggingFace has become the central platform for transfer learning. Understanding its ecosystem is as essential as understanding Git for a software engineer.

Complete Fine-Tuning Pipeline with HuggingFace

Python
# Complete BERT Hindi Sentiment Fine-Tuning Pipeline
# This is a real, production-ready example

from transformers import (
    AutoTokenizer, AutoModelForSequenceClassification,
    TrainingArguments, Trainer
)
from datasets import load_dataset
import numpy as np
from sklearn.metrics import accuracy_score, f1_score

# ─── Step 1: Load Indian language model ───
model_name = "ai4bharat/indic-bert"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
    model_name, num_labels=2  # Positive / Negative
)

# ─── Step 2: Load Hindi sentiment dataset ───
dataset = load_dataset("ai4bharat/IndicSentiment", "hi")

# ─── Step 3: Tokenize ───
def tokenize_fn(examples):
    return tokenizer(
        examples["text"], 
        padding="max_length", 
        truncation=True, 
        max_length=128
    )

tokenized = dataset.map(tokenize_fn, batched=True)

# ─── Step 4: Define metrics ───
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    return {
        "accuracy": accuracy_score(labels, preds),
        "f1": f1_score(labels, preds, average="weighted")
    }

# ─── Step 5: Training arguments ───
training_args = TrainingArguments(
    output_dir="./hindi-sentiment-model",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    learning_rate=2e-5,           # Small LR for fine-tuning!
    weight_decay=0.01,
    warmup_ratio=0.1,             # Warm up for 10% of training
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    fp16=True,                    # Mixed precision — 2× speed
    logging_steps=50,
)

# ─── Step 6: Train! ───
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized["train"],
    eval_dataset=tokenized["validation"],
    compute_metrics=compute_metrics,
)

trainer.train()

# ─── Step 7: Evaluate ───
results = trainer.evaluate()
print(f"Hindi Sentiment Accuracy: {results['eval_accuracy']:.3f}")
print(f"Hindi Sentiment F1: {results['eval_f1']:.3f}")

LoRA Fine-Tuning with HuggingFace PEFT

Python
# LoRA fine-tuning — same task, 100× fewer trainable params
from peft import LoraConfig, get_peft_model, TaskType

# ─── Configure LoRA ───
lora_config = LoraConfig(
    task_type=TaskType.SEQ_CLS,     # Sequence classification
    r=8,                             # LoRA rank
    lora_alpha=16,                   # Scaling factor
    lora_dropout=0.1,                # Dropout for regularization
    target_modules=["query", "value"],  # Apply LoRA to attention layers
)

# ─── Wrap model with LoRA ───
model = AutoModelForSequenceClassification.from_pretrained(
    "ai4bharat/indic-bert", num_labels=2
)
peft_model = get_peft_model(model, lora_config)

# ─── Check parameter reduction ───
peft_model.print_trainable_parameters()
# Output: trainable params: 294,914 || all params: 33,651,458 || 
#         trainable%: 0.8765%

# ─── Train exactly the same way! ───
trainer = Trainer(
    model=peft_model,
    args=training_args,
    train_dataset=tokenized["train"],
    eval_dataset=tokenized["validation"],
    compute_metrics=compute_metrics,
)
trainer.train()

Worked Examples

Example 1: By-Hand — Computing LoRA Parameter Savings

Solution:

Step 1: Target matrix parameters

Each Q or V matrix: 768 × 768 = 589,824 parameters

Per layer: 2 matrices × 589,824 = 1,179,648 parameters

All 12 layers: 12 × 1,179,648 = 14,155,776 target parameters

Step 2: LoRA parameters per matrix

For each target matrix (768 × 768), LoRA adds:

B ∈ ℝ^768×4 = 3,072 params + A ∈ ℝ^4×768 = 3,072 params = 6,144 per matrix

Per layer: 2 × 6,144 = 12,288

All 12 layers: 12 × 12,288 = 147,456 LoRA parameters

Step 3: Reduction ratio

Step 4: BERT-base has ~110M total parameters

Trainable: 147,456 + ~1,538 (classifier head for 2 classes) ≈ 149,000

Percentage: 149,000 / 110,000,000 = 0.14% trainable

Example 2: Indian Industry — Flipkart Product Categorization

Example 3: US/Global — HuggingFace Model Hub Workflow

Python
# Multi-language sentiment — 4 languages, 1 model
from transformers import pipeline

# XLM-RoBERTa: pretrained on 100 languages!
classifier = pipeline(
    "sentiment-analysis",
    model="cardiffnlp/twitter-xlm-roberta-base-sentiment"
)

# It just works — for ALL languages
texts = [
    "This product is amazing!",         # English
    "Este producto es terrible",         # Spanish
    "यह प्रोडक्ट बहुत अच्छा है",           # Hindi
    "Este produto é horrível",           # Portuguese
]

for text in texts:
    result = classifier(text)[0]
    print(f"{result['label']:8s} ({result['score']:.3f}): {text}")

From-Scratch NumPy Implementation

Transfer Learning Simulator: Feature Extraction vs. Fine-Tuning

NumPy
import numpy as np

"""
Transfer Learning Simulator — From Scratch
==========================================
We simulate a 3-layer neural network:
  Layer 1: "Universal features" (edges, textures)
  Layer 2: "Mid-level features" (object parts)
  Layer 3: "Task-specific classifier"

Phase 1: Pretrain on "source task" (1000 samples, 10 classes)
Phase 2: Transfer to "target task" (100 samples, 5 classes)
Compare: Feature extraction vs Fine-tuning vs From scratch
"""

np.random.seed(42)

# ─── Helper functions ───
def relu(x):
    return np.maximum(0, x)

def relu_grad(x):
    return (x > 0).astype(float)

def softmax(x):
    e = np.exp(x - x.max(axis=1, keepdims=True))
    return e / e.sum(axis=1, keepdims=True)

def cross_entropy(probs, labels):
    n = labels.shape[0]
    return -np.log(probs[range(n), labels] + 1e-8).mean()

class SimpleNet:
    """3-layer network for transfer learning demonstration"""
    
    def __init__(self, input_dim, hidden1, hidden2, output_dim):
        scale = 0.01
        self.W1 = np.random.randn(input_dim, hidden1) * scale
        self.b1 = np.zeros((1, hidden1))
        self.W2 = np.random.randn(hidden1, hidden2) * scale
        self.b2 = np.zeros((1, hidden2))
        self.W3 = np.random.randn(hidden2, output_dim) * scale
        self.b3 = np.zeros((1, output_dim))
    
    def forward(self, X):
        self.z1 = X @ self.W1 + self.b1
        self.a1 = relu(self.z1)
        self.z2 = self.a1 @ self.W2 + self.b2
        self.a2 = relu(self.z2)
        self.z3 = self.a2 @ self.W3 + self.b3
        self.probs = softmax(self.z3)
        self.X = X
        return self.probs
    
    def backward(self, labels, freeze_layers=None):
        """
        freeze_layers: set of layer indices to freeze {1, 2}
        """
        if freeze_layers is None:
            freeze_layers = set()
        
        n = labels.shape[0]
        
        # Output gradient
        dz3 = self.probs.copy()
        dz3[range(n), labels] -= 1
        dz3 /= n
        
        # Layer 3 gradients (classifier — always train)
        self.dW3 = self.a2.T @ dz3
        self.db3 = dz3.sum(axis=0, keepdims=True)
        
        # Layer 2 gradients
        da2 = dz3 @ self.W3.T
        dz2 = da2 * relu_grad(self.z2)
        if 2 not in freeze_layers:
            self.dW2 = self.a1.T @ dz2
            self.db2 = dz2.sum(axis=0, keepdims=True)
        else:
            self.dW2 = np.zeros_like(self.W2)  # FROZEN!
            self.db2 = np.zeros_like(self.b2)
        
        # Layer 1 gradients
        da1 = dz2 @ self.W2.T
        dz1 = da1 * relu_grad(self.z1)
        if 1 not in freeze_layers:
            self.dW1 = self.X.T @ dz1
            self.db1 = dz1.sum(axis=0, keepdims=True)
        else:
            self.dW1 = np.zeros_like(self.W1)  # FROZEN!
            self.db1 = np.zeros_like(self.b1)
    
    def update(self, lr):
        self.W1 -= lr * self.dW1
        self.b1 -= lr * self.db1
        self.W2 -= lr * self.dW2
        self.b2 -= lr * self.db2
        self.W3 -= lr * self.dW3
        self.b3 -= lr * self.db3
    
    def accuracy(self, X, y):
        probs = self.forward(X)
        preds = np.argmax(probs, axis=1)
        return (preds == y).mean()
    
    def replace_head(self, new_output_dim):
        """Replace classifier for new task"""
        old_hidden = self.W3.shape[0]
        self.W3 = np.random.randn(old_hidden, new_output_dim) * 0.01
        self.b3 = np.zeros((1, new_output_dim))
    
    def copy_backbone_from(self, other):
        """Copy layers 1 and 2 from another network"""
        self.W1 = other.W1.copy()
        self.b1 = other.b1.copy()
        self.W2 = other.W2.copy()
        self.b2 = other.b2.copy()

# ─── Generate synthetic data ───
def make_data(n_samples, n_features, n_classes, seed=0):
    np.random.seed(seed)
    X = np.random.randn(n_samples, n_features)
    y = np.random.randint(0, n_classes, n_samples)
    # Add structure: make classes slightly separable
    for c in range(n_classes):
        mask = y == c
        X[mask] += np.random.randn(n_features) * 0.5
    return X, y

# Source task (large, like ImageNet)
X_source, y_source = make_data(1000, 20, 10, seed=1)
# Target task (small, like Flipkart products)
X_target_train, y_target_train = make_data(100, 20, 5, seed=2)
X_target_test, y_target_test = make_data(200, 20, 5, seed=3)

# ─── Phase 1: Pretrain on source ───
print("═══ Phase 1: Pretraining on Source Task ═══")
pretrained = SimpleNet(20, 64, 32, 10)
for epoch in range(200):
    pretrained.forward(X_source)
    pretrained.backward(y_source)
    pretrained.update(lr=0.1)
print(f"Source accuracy: {pretrained.accuracy(X_source, y_source):.3f}")

# ─── Approach 1: From Scratch ───
print("\n═══ Approach 1: Training from Scratch ═══")
scratch = SimpleNet(20, 64, 32, 5)
for epoch in range(200):
    scratch.forward(X_target_train)
    scratch.backward(y_target_train)
    scratch.update(lr=0.1)
scratch_acc = scratch.accuracy(X_target_test, y_target_test)
print(f"From-scratch test accuracy: {scratch_acc:.3f}")

# ─── Approach 2: Feature Extraction (freeze layers 1 & 2) ───
print("\n═══ Approach 2: Feature Extraction (Frozen Backbone) ═══")
feat_extract = SimpleNet(20, 64, 32, 5)
feat_extract.copy_backbone_from(pretrained)
for epoch in range(200):
    feat_extract.forward(X_target_train)
    feat_extract.backward(y_target_train, freeze_layers={1, 2})
    feat_extract.update(lr=0.1)
feat_acc = feat_extract.accuracy(X_target_test, y_target_test)
print(f"Feature extraction test accuracy: {feat_acc:.3f}")

# ─── Approach 3: Full Fine-Tuning ───
print("\n═══ Approach 3: Full Fine-Tuning ═══")
fine_tuned = SimpleNet(20, 64, 32, 5)
fine_tuned.copy_backbone_from(pretrained)
for epoch in range(200):
    fine_tuned.forward(X_target_train)
    fine_tuned.backward(y_target_train)  # No freezing!
    fine_tuned.update(lr=0.01)  # Smaller LR for fine-tuning
ft_acc = fine_tuned.accuracy(X_target_test, y_target_test)
print(f"Fine-tuned test accuracy: {ft_acc:.3f}")

# ─── Summary ───
print("\n═══ RESULTS COMPARISON ═══")
print(f"From scratch:       {scratch_acc:.3f}")
print(f"Feature extraction: {feat_acc:.3f}")
print(f"Fine-tuning:        {ft_acc:.3f}")

PyTorch / HuggingFace Production Code

Complete ResNet50 Fine-Tuning for Indian Products

PyTorch
"""
ResNet50 Fine-Tuning Pipeline — Flipkart Product Classification
===============================================================
Classifies: saree, kurta, jeans, chai, smartphone, jhumka, etc.
Uses discriminative learning rates + gradual unfreezing
"""
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import models, transforms, datasets
from torch.utils.data import DataLoader

# ─── Config ───
NUM_CLASSES = 50
BATCH_SIZE = 32
BASE_LR = 3e-4
EPOCHS = 15
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# ─── Data transforms ───
train_transform = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(brightness=0.2, contrast=0.2),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], 
                         [0.229, 0.224, 0.225])
])

val_transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406],
                         [0.229, 0.224, 0.225])
])

# ─── Load data (ImageFolder structure) ───
# Expects: data/train/saree/*, data/train/kurta/*, etc.
train_dataset = datasets.ImageFolder("data/train", train_transform)
val_dataset = datasets.ImageFolder("data/val", val_transform)
train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, 
                          shuffle=True, num_workers=4)
val_loader = DataLoader(val_dataset, batch_size=BATCH_SIZE*2,
                        num_workers=4)

# ─── Build model ───
model = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)

# Replace classifier head
model.fc = nn.Sequential(
    nn.Dropout(0.3),
    nn.Linear(2048, 512),
    nn.ReLU(),
    nn.Dropout(0.2),
    nn.Linear(512, NUM_CLASSES)
)
model = model.to(DEVICE)

# ─── Discriminative learning rates ───
param_groups = [
    {"params": model.conv1.parameters(),  "lr": BASE_LR/100},
    {"params": model.bn1.parameters(),    "lr": BASE_LR/100},
    {"params": model.layer1.parameters(), "lr": BASE_LR/50},
    {"params": model.layer2.parameters(), "lr": BASE_LR/10},
    {"params": model.layer3.parameters(), "lr": BASE_LR/5},
    {"params": model.layer4.parameters(), "lr": BASE_LR},
    {"params": model.fc.parameters(),     "lr": BASE_LR*3},
]

optimizer = optim.AdamW(param_groups, weight_decay=1e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=EPOCHS)
criterion = nn.CrossEntropyLoss(label_smoothing=0.1)

# ─── Gradual unfreezing ───
freeze_schedule = {
    0: ["conv1", "bn1", "layer1", "layer2", "layer3", "layer4"],  # Freeze all backbone
    3: ["conv1", "bn1", "layer1", "layer2", "layer3"],              # Unfreeze layer4
    6: ["conv1", "bn1", "layer1", "layer2"],                        # Unfreeze layer3
    9: ["conv1", "bn1", "layer1"],                                    # Unfreeze layer2
    12: [],                                                             # Everything unfrozen
}

def apply_freeze(model, frozen_names):
    for name, module in model.named_children():
        for param in module.parameters():
            param.requires_grad = name not in frozen_names

# ─── Training loop ───
for epoch in range(EPOCHS):
    # Apply freeze schedule
    if epoch in freeze_schedule:
        apply_freeze(model, freeze_schedule[epoch])
        trainable = sum(p.numel() for p in model.parameters() 
                        if p.requires_grad)
        print(f"Epoch {epoch}: {trainable:,} trainable params")
    
    # Train
    model.train()
    running_loss = 0
    for images, labels in train_loader:
        images, labels = images.to(DEVICE), labels.to(DEVICE)
        optimizer.zero_grad()
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    
    # Evaluate
    model.eval()
    correct = total = 0
    with torch.no_grad():
        for images, labels in val_loader:
            images, labels = images.to(DEVICE), labels.to(DEVICE)
            outputs = model(images)
            _, predicted = outputs.max(1)
            total += labels.size(0)
            correct += predicted.eq(labels).sum().item()
    
    val_acc = correct / total
    scheduler.step()
    print(f"Epoch {epoch+1}/{EPOCHS} | "
          f"Loss: {running_loss/len(train_loader):.4f} | "
          f"Val Acc: {val_acc:.4f}")

# ─── Save model ───
torch.save(model.state_dict(), "flipkart_product_classifier.pth")

Visual Diagrams

Diagram 1: Transfer Learning Taxonomy

Diagram 2: Feature Extraction vs. Fine-Tuning Pipeline

Diagram 3: LoRA vs. Full Fine-Tuning Memory Footprint

Industry Case Study: Flipkart India 🇮🇳

Architecture
ImageNet-pretrained ResNet50
    │
    ├── Conv1-Layer3: FROZEN (universal visual features)
    │   ├── Edges, textures, colors → Detect fabric weave
    │   ├── Corners, patterns → Detect zari/border designs  
    │   └── Object parts → Detect pallu, pleats
    │
    ├── Layer4: FINE-TUNED (lr=1e-4)
    │   └── Adapted to Indian product parts
    │
    └── New Head: TRAINED (lr=1e-3)
        ├── Dropout(0.3)
        ├── Linear(2048 → 512) + ReLU
        ├── Dropout(0.2)
        └── Linear(512 → 80)  # 80 product categories

Industry Case Study: HuggingFace USA 🇺🇸

# Before HuggingFace: 100+ lines of model loading code
# After HuggingFace: 2 lines
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("bert-base-uncased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

from transformers import pipeline

# Sentiment analysis — uses fine-tuned DistilBERT
sentiment = pipeline("sentiment-analysis")
sentiment("I love this product!")
# [{'label': 'POSITIVE', 'score': 0.9998}]

# Translation — uses fine-tuned MarianMT
translator = pipeline("translation_en_to_hi", model="Helsinki-NLP/opus-mt-en-hi")
translator("Transfer learning is powerful")
# [{'translation_text': 'स्थानांतरण सीखना शक्तिशाली है'}]

# Zero-shot classification — uses fine-tuned BART
classifier = pipeline("zero-shot-classification")
classifier(
    "The stock market crashed today",
    candidate_labels=["finance", "sports", "technology"]
)
# {'labels': ['finance', 'technology', 'sports'], 
#  'scores': [0.95, 0.03, 0.02]}

Common Misconceptions

GATE / Exam Corner

GATE-Style MCQs

Interview Prep

Conceptual Questions

Coding Challenge

import torch
import torchvision.models as models

model = models.resnet18(pretrained=True)

# Bug 1: Freeze backbone
for param in model.parameters():
    param.requires_grad = True  # 🐛

# Bug 2: Replace head
model.classifier = nn.Linear(512, 10)  # 🐛

# Bug 3: Optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)  # 🐛

System Design: Indian Language Sentiment System

Customer Review (any language)
        │
        ▼
┌─────────────────┐
│ Language Detect  │ ← fastText langid (99.5% acc for Indian langs)
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ XLM-RoBERTa or  │ ← Single multilingual model
│ IndicBERT       │   Fine-tuned on combined Hi+Ta+Bn data
│ + LoRA adapters │   Separate LoRA per language (optional)
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Sentiment       │ ← {positive, negative, neutral}
│ + Aspect        │ ← {delivery, quality, price, service}
└─────────────────┘

Hands-On Lab / Mini-Project

Exercises

Section A: Conceptual (5 Questions)

Section B: Mathematical (8 Questions)

Section C: Coding (4 Questions)

Section D: Critical Thinking (3 Questions)

★ Starred Research Questions (2)

Connections

Chapter Summary

Further Reading

🇮🇳 Indian Resources

• NPTEL: "Deep Learning" by Prof. Mitesh Khapra (IIT Madras) — Lecture 37-39 cover transfer learning with Indian language examples
• AI4Bharat: ai4bharat.org — IndicBERT, IndicTrans, IndicNLP models and papers
• GATE: Refer to Goodfellow, Bengio & Courville, Chapter 15.2 (Transfer Learning). GATE DA 2024 syllabus includes "Transfer Learning fundamentals"
• IIT Madras CS6910: Course on Deep Learning — Assignments on fine-tuning for Indian language tasks

🌍 Global Resources

• Original Papers:
  • Yosinski et al. (2014): "How transferable are features in deep neural networks?" — The foundational paper on layer transferability
  • Howard & Ruder (2018): "Universal Language Model Fine-tuning for Text Classification" (ULMFiT) — Introduced gradual unfreezing and discriminative LR
  • Hu et al. (2021): "LoRA: Low-Rank Adaptation of Large Language Models" — The LoRA paper
  • Dettmers et al. (2023): "QLoRA: Efficient Finetuning of Quantized Language Models"
• HuggingFace Course: huggingface.co/learn — Free, comprehensive, with Colab notebooks
• fast.ai: Practical Deep Learning for Coders — Jeremy Howard's course emphasizes transfer learning throughout
• Distill.pub: "Feature Visualization" — Interactive exploration of what CNN layers learn
• 3Blue1Brown: "Neural Networks" series — Visual intuition for why features transfer

📚 Books

• Chollet, "Deep Learning with Python" (2nd ed.) — Chapter 8 covers transfer learning with Keras
• Goodfellow, Bengio & Courville, "Deep Learning" — Chapter 15 for theoretical foundations
• Tunstall, von Werra & Wolf, "Natural Language Processing with Transformers" — The HuggingFace companion book