Deep Learning

Deep Learning uses multi-layered neural networks to learn representations of data with multiple levels of abstraction. It drives modern AI advances in computer vision, NLP, and more.

Neural Networks

A neural network is a chain of layers, each containing neurons. Every neuron computes a weighted sum of its inputs, adds a bias, then passes the result through a non-linear activation function:

Stacking many such layers allows the network to learn increasingly abstract representations — edges → shapes → objects in a vision model, or tokens → phrases → semantics in a language model.

The Perceptron

The simplest unit: a single neuron with a step activation. Not learnable for non-linear problems, but the building block for everything else.

import numpy as np

def perceptron(x, w, b):
    return 1 if np.dot(w, x) + b >= 0 else 0

# XOR is NOT linearly separable — a single perceptron fails here
x = np.array([1, 0])
w = np.array([1, 1])
b = -1.5
print(perceptron(x, w, b))  # 0

Feed-Forward Network (NumPy)

A two-layer network doing a forward pass manually — this is exactly what frameworks do under the hood:

import numpy as np

def relu(z):
    return np.maximum(0, z)

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

# Random weights for a 3 → 4 → 2 network
np.random.seed(0)
W1 = np.random.randn(4, 3) * 0.1   # hidden layer weights
b1 = np.zeros(4)
W2 = np.random.randn(2, 4) * 0.1   # output layer weights
b2 = np.zeros(2)

x = np.array([0.5, -0.3, 0.8])     # one input sample

# Forward pass
h = relu(W1 @ x + b1)              # hidden activations
y = softmax(W2 @ h + b2)           # class probabilities
print("Probabilities:", y)          # e.g. [0.52, 0.48]

Activation Functions

Without non-linearity, stacking layers is equivalent to a single linear transformation. Activations introduce the non-linearity that lets deep networks approximate any function.

import torch
import torch.nn.functional as F

x = torch.tensor([-2.0, 0.0, 1.5, 3.0])

print("ReLU:   ", F.relu(x))
print("Sigmoid:", torch.sigmoid(x))
print("Tanh:   ", torch.tanh(x))
print("Softmax:", F.softmax(x, dim=0))

Training: Loss & Optimisers

Training a neural network is an optimisation problem: minimise a loss function that measures how wrong the predictions are, by adjusting weights via backpropagation and an optimiser.

Common Loss Functions

Backpropagation in One Sentence

The chain rule of calculus applied from the loss backwards through each layer — computing ∂Loss/∂w for every weight so the optimiser knows which direction to nudge each parameter.

Optimisers

import torch.optim as optim

# SGD with momentum — simple, effective, widely used
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

# Adam — adaptive learning rates per parameter, default choice for most problems
optimizer = optim.Adam(model.parameters(), lr=1e-3)

# One training step
optimizer.zero_grad()   # clear accumulated gradients
loss.backward()         # compute gradients via backprop
optimizer.step()        # update weights: w = w - lr * grad

Regularisation

Deep networks easily memorise training data. Regularisation techniques force the model to generalise.

Dropout

Randomly zeroes a fraction of neurons during each training step. Forces redundant representations — the network can't rely on any single neuron.

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(128, 64),
    nn.ReLU(),
    nn.Dropout(p=0.5),   # 50% of neurons zeroed each forward pass during training
    nn.Linear(64, 10),
)

Batch Normalisation

Normalises each mini-batch's activations to zero mean and unit variance, then learns a scale and shift. Stabilises training and allows higher learning rates.

model = nn.Sequential(
    nn.Linear(128, 64),
    nn.BatchNorm1d(64),  # normalise activations across the batch
    nn.ReLU(),
    nn.Linear(64, 10),
)

Early Stopping

Monitor validation loss during training and stop when it stops improving — the simplest and often most effective regulariser.

best_val_loss = float('inf')
patience, wait = 5, 0

for epoch in range(100):
    train(model)
    val_loss = evaluate(model)
    if val_loss < best_val_loss:
        best_val_loss = val_loss
        wait = 0
        torch.save(model.state_dict(), 'best_model.pt')
    else:
        wait += 1
        if wait >= patience:
            print(f"Early stopping at epoch {epoch}")
            break

Full PyTorch Training Loop

Putting it all together — a complete classifier on the Iris dataset using PyTorch:

import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Data
X, y = load_iris(return_X_y=True)
X = StandardScaler().fit_transform(X)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

X_train = torch.tensor(X_train, dtype=torch.float32)
X_val   = torch.tensor(X_val,   dtype=torch.float32)
y_train = torch.tensor(y_train, dtype=torch.long)
y_val   = torch.tensor(y_val,   dtype=torch.long)

# Model
class IrisNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(4, 32),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(32, 16),
            nn.ReLU(),
            nn.Linear(16, 3),
        )
    def forward(self, x):
        return self.net(x)

model     = IrisNet()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

# Training loop
for epoch in range(100):
    model.train()
    optimizer.zero_grad()
    loss = criterion(model(X_train), y_train)
    loss.backward()
    optimizer.step()

    if (epoch + 1) % 20 == 0:
        model.eval()
        with torch.no_grad():
            preds = model(X_val).argmax(dim=1)
            acc = (preds == y_val).float().mean()
        print(f"Epoch {epoch+1:3d} | loss {loss:.4f} | val acc {acc:.2%}")

# Epoch  20 | loss 0.6234 | val acc 93.33%
# Epoch  40 | loss 0.3891 | val acc 96.67%
# Epoch  60 | loss 0.2754 | val acc 96.67%
# Epoch  80 | loss 0.2103 | val acc 96.67%
# Epoch 100 | loss 0.1782 | val acc 100.00%

Gradient Descent Visualizer

See how gradient descent navigates a loss landscape. Click the curve to place the ball, adjust the learning rate, then step through the optimisation.

Activation Function Playground

Compare activation functions side by side. Toggle functions on/off and drag the scrubber to read off exact values.

Projects

Hand Gesture Recognition

CV Backend Req.

Zero

One

Two

Ok

Up

Down

Knowledge Check

Test your understanding — pick the best answer, then click Check.