Deep Learning

Deep Learning uses multi-layered neural networks to learn representations of data with multiple levels of abstraction. It drives modern AI advances in computer vision, NLP, and more.

Neural Networks

A neural network is a chain of layers, each containing neurons. Every neuron computes a weighted sum of its inputs, adds a bias, then passes the result through a non-linear activation function:

Stacking many such layers allows the network to learn increasingly abstract representations — edges → shapes → objects in a vision model, or tokens → phrases → semantics in a language model.

Neural networks are loosely modelled on biological neurons and the nervous system. The architecture is a large number of simple processing nodes (neurons) organised in layers. Each node connects to several nodes in the previous and next layer: it receives inputs, processes them and passes its output on.

Parts of a Neural Network

• Input layer: accepts the data. It has one neuron per feature your network uses to make predictions.
• Hidden layers: every layer between input and output.
• Output layer: produces the prediction. For regression it has one neuron per predicted value. For binary classification, one output neuron. For multi-class classification, one neuron per class.

The number of hidden layers and neurons depends on the problem, and finding a good setup is largely trial and error. Using the same number of neurons in every hidden layer is usually enough. Sometimes a large first layer followed by smaller ones works better, because the first layer learns many low-level features that feed a few higher-order ones. As a rule of thumb, 1 to 5 hidden layers with 1 to 100 neurons each serves most problems. Start there and add capacity until the model starts to overfit. Too few layers or neurons and the model cannot learn the patterns in your data at all.

The Perceptron

The simplest unit: a single neuron with a step activation. Not learnable for non-linear problems, but the building block for everything else.

import numpy as np

def perceptron(x, w, b):
    return 1 if np.dot(w, x) + b >= 0 else 0

# XOR is NOT linearly separable — a single perceptron fails here
x = np.array([1, 0])
w = np.array([1, 1])
b = -1.5
print(perceptron(x, w, b))  # 0

Feed-Forward Network (NumPy)

A two-layer network doing a forward pass manually — this is exactly what frameworks do under the hood:

import numpy as np

def relu(z):
    return np.maximum(0, z)

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

# Random weights for a 3 → 4 → 2 network
np.random.seed(0)
W1 = np.random.randn(4, 3) * 0.1   # hidden layer weights
b1 = np.zeros(4)
W2 = np.random.randn(2, 4) * 0.1   # output layer weights
b2 = np.zeros(2)

x = np.array([0.5, -0.3, 0.8])     # one input sample

# Forward pass
h = relu(W1 @ x + b1)              # hidden activations
y = softmax(W2 @ h + b2)           # class probabilities
print("Probabilities:", y)          # e.g. [0.52, 0.48]

Hyperparameters

Hyperparameters are settings you choose before training rather than values the network learns. Three matter in almost every project.

Batch Size

The number of training examples used in one iteration, i.e. how many samples are processed before the model parameters are updated. A dataset can be split into one or more batches:

• Batch gradient descent: batch size equals the size of the training set.
• Stochastic gradient descent: batch size is 1.
• Mini-batch gradient descent: batch size is between 1 and the training set size. Popular choices are 32, 64 and 128.

Number of Epochs

An epoch is one full pass through the training dataset, in which every sample is used to update the parameters (usually split across several batches). The learning algorithm typically runs for many epochs, often hundreds or thousands, until the error is sufficiently reduced.

Learning Rate

The learning rate sets the size of the weight adjustment at each iteration, based on the estimated error. It is the step size of training. Too high, and the model may jump to a poor solution or overshoot. Too low, and optimisation crawls or gets stuck. Pick a value that balances speed of convergence against the quality of the solution.

Activation Functions

Without non-linearity, stacking layers is equivalent to a single linear transformation. Activations introduce the non-linearity that lets deep networks approximate any function.

As a rough guide, hidden-layer performance tends to improve in this order, from lowest to highest: logistic, tanh, ReLU, Leaky ReLU, ELU, SELU.

Output Layer Activation

• Regression: usually no activation, since the output can take any value. To bound the output, use tanh for [−1, 1] or sigmoid for [0, 1]. For strictly positive outputs, use softplus.
• Binary classification: sigmoid, so the output lies between 0 and 1.
• Multi-class classification: softmax, so the class probabilities sum to 1.

import torch
import torch.nn.functional as F

x = torch.tensor([-2.0, 0.0, 1.5, 3.0])

print("ReLU:   ", F.relu(x))
print("Sigmoid:", torch.sigmoid(x))
print("Tanh:   ", torch.tanh(x))
print("Softmax:", F.softmax(x, dim=0))

Training: Loss & Optimisers

Training a neural network is an optimisation problem: minimise a loss function that measures how wrong the predictions are, by adjusting weights via backpropagation and an optimiser.

Loss vs Cost

The two terms are often used interchangeably, but there is a difference. A loss is computed for a single training example. A cost is the average loss over the whole training set. Training minimises it.

Common Loss Functions

Backpropagation in One Sentence

The chain rule of calculus applied from the loss backwards through each layer — computing ∂Loss/∂w for every weight so the optimiser knows which direction to nudge each parameter.

Optimisers

An optimiser is the algorithm that adjusts the model's parameters to minimise the loss. The choice can noticeably affect performance.

Gradient Descent

Iteratively move the parameters in the direction that reduces the loss: compute the gradient of the loss with respect to the parameters and step the opposite way. The step size is the learning rate:

$$b = a - \gamma \nabla f(a)$$

where \(b\) is the next position, \(a\) the current position, \(\gamma\) the learning rate and \(\nabla f(a)\) the gradient of the loss. If the learning rate is too high, the algorithm overshoots the minimum and oscillates around it. If it is too low, convergence is slow. Batch, mini-batch and stochastic variants trade off speed, noise and memory differently.

Adam

Adam (Adaptive Moment Estimation) combines SGD with momentum and adapts the learning rate for each weight. It keeps an exponentially decaying average of past gradients to scale each parameter's step, so the learning rate adapts to the data and the model. Momentum adds a fraction of the previous update to the current one, which smooths oscillations and helps escape local minima and saddle points. The result usually converges faster and more robustly than plain SGD.

import torch.optim as optim

# SGD with momentum — simple, effective, widely used
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

# Adam — adaptive learning rates per parameter, default choice for most problems
optimizer = optim.Adam(model.parameters(), lr=1e-3)

# One training step
optimizer.zero_grad()   # clear accumulated gradients
loss.backward()         # compute gradients via backprop
optimizer.step()        # update weights: w = w - lr * grad

Regularisation

Deep networks easily memorise training data. Regularisation techniques force the model to generalise.

L1 and L2 Regularisation

Both add a penalty term to the loss during training, scaled by a regularisation rate \(\lambda\).

• L1 penalises the sum of the absolute values of the weights, pushing many weights towards zero.
• L2, also called weight decay, penalises the sum of the squared weights, encouraging smaller weights so the model does not overfit the training data.

$$\text{Cost} = \text{Loss} + \frac{\lambda}{2m} \sum \lVert w \rVert$$

$$\text{Cost} = \text{Loss} + \frac{\lambda}{2m} \sum \lVert w \rVert^2$$

Dropout

Randomly zeroes a fraction of neurons during each training step. Forces redundant representations — the network can't rely on any single neuron, so knowledge is distributed across the whole network. A good rate is between 0.1 and 0.5: about 0.3 for RNNs and 0.5 for CNNs, with larger rates for bigger layers. Raising the rate reduces overfitting. Lowering it helps against underfitting.

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(128, 64),
    nn.ReLU(),
    nn.Dropout(p=0.5),   # 50% of neurons zeroed each forward pass during training
    nn.Linear(64, 10),
)

Batch Normalisation

Normalises each mini-batch's activations to zero mean and unit variance, then learns a scale and shift. Stabilises training and allows higher learning rates.

model = nn.Sequential(
    nn.Linear(128, 64),
    nn.BatchNorm1d(64),  # normalise activations across the batch
    nn.ReLU(),
    nn.Linear(64, 10),
)

Early Stopping

Monitor validation loss during training and stop when it stops improving — the simplest and often most effective regulariser. The number of epochs to wait before stopping is a hyperparameter to tune (called patience in Keras). It is often combined with weight decay and dropout.

best_val_loss = float('inf')
patience, wait = 5, 0

for epoch in range(100):
    train(model)
    val_loss = evaluate(model)
    if val_loss < best_val_loss:
        best_val_loss = val_loss
        wait = 0
        torch.save(model.state_dict(), 'best_model.pt')
    else:
        wait += 1
        if wait >= patience:
            print(f"Early stopping at epoch {epoch}")
            break

Full PyTorch Training Loop

Putting it all together — a complete classifier on the Iris dataset using PyTorch:

import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Data
X, y = load_iris(return_X_y=True)
X = StandardScaler().fit_transform(X)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

X_train = torch.tensor(X_train, dtype=torch.float32)
X_val   = torch.tensor(X_val,   dtype=torch.float32)
y_train = torch.tensor(y_train, dtype=torch.long)
y_val   = torch.tensor(y_val,   dtype=torch.long)

# Model
class IrisNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(4, 32),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(32, 16),
            nn.ReLU(),
            nn.Linear(16, 3),
        )
    def forward(self, x):
        return self.net(x)

model     = IrisNet()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

# Training loop
for epoch in range(100):
    model.train()
    optimizer.zero_grad()
    loss = criterion(model(X_train), y_train)
    loss.backward()
    optimizer.step()

    if (epoch + 1) % 20 == 0:
        model.eval()
        with torch.no_grad():
            preds = model(X_val).argmax(dim=1)
            acc = (preds == y_val).float().mean()
        print(f"Epoch {epoch+1:3d} | loss {loss:.4f} | val acc {acc:.2%}")

# Epoch  20 | loss 0.6234 | val acc 93.33%
# Epoch  40 | loss 0.3891 | val acc 96.67%
# Epoch  60 | loss 0.2754 | val acc 96.67%
# Epoch  80 | loss 0.2103 | val acc 96.67%
# Epoch 100 | loss 0.1782 | val acc 100.00%

Gradient Descent Visualizer

See how gradient descent navigates a loss landscape. Click the curve to place the ball, adjust the learning rate, then step through the optimisation.

Activation Function Playground

Compare activation functions side by side. Toggle functions on/off and drag the scrubber to read off exact values.

Projects

Hand Gesture Recognition

CV Backend Req.

Zero

One

Two

Ok

Up

Down