Computer Vision

Computer Vision enables machines to interpret and understand visual information from the world. At its core is the Convolutional Neural Network (CNN) — an architecture specifically designed to process grid-like data such as images.

The Convolution Operation

A convolution slides a small filter (kernel) across an image, computing a dot product at each position. This produces a feature map — a compact representation that highlights edges, textures, or patterns the filter was designed to detect.

import numpy as np
from scipy.signal import convolve2d

image = np.array([
    [1,1,1,0,0],
    [0,1,1,1,0],
    [0,0,1,1,1],
    [0,0,1,1,0],
    [0,1,1,0,0]
])

# Edge-detection filter (Sobel-like)
kernel = np.array([
    [1, 0, 1],
    [0, 1, 0],
    [1, 0, 1]
])

feature_map = convolve2d(image, kernel, mode='valid')
print(feature_map)

How a convolution layer works

A convolution layer needs three things: input data, a filter and a feature map. The filter (also called a kernel or feature detector) is a small 2-D array of weights that represents part of the image. It is usually 3×3, which also sets the size of the receptive field. The filter is applied to one area of the image, a dot product is taken between the input pixels and the filter, and the result goes into an output array. The filter then shifts by a stride and repeats until it has swept the whole image. The final output is the feature map (also called an activation map or convolved feature).

Three hyperparameters set the size of the output and must be chosen before training:

• Number of filters sets the depth of the output. Three distinct filters give three feature maps, so a depth of three.
• Stride is the number of pixels the kernel moves each step. Values of two or more are rare, and a larger stride gives a smaller output.
• Zero-padding is used when the filter does not fit the image, by treating everything outside the input as zero. There are three types:
  • Valid (no padding): the last convolution is dropped if the dimensions do not line up.
  • Same: the output has the same size as the input.
  • Full: zeros are added around the border, so the output is larger than the input.

After each convolution, a CNN applies a ReLU transformation to the feature map, which adds non-linearity. Because an output value does not depend on every input pixel, convolutional and pooling layers are called "partially connected". In effect, the convolution layer turns the image into numbers the network can interpret and extract patterns from.

Filter Effect Gallery

Apply classic image filters to a small pixel canvas. Draw your own pattern, then see how each kernel transforms it — edge detection, blur, sharpening, and emboss.

CNN Architecture

A CNN stacks several types of layers. Each layer learns progressively more abstract features — from edges in early layers to shapes and objects in later layers.

Pooling layers in more detail

Pooling (downsampling) reduces dimensionality and so the number of parameters. Like a convolution, the pooling filter sweeps across the input, but it has no weights. It applies an aggregation function to the values in its receptive field instead.

• Max pooling sends the largest value in each window to the output. It is used more often than average pooling.
• Average pooling sends the mean of each window.

A lot of information is lost in pooling, but it reduces complexity, improves efficiency and limits the risk of overfitting.

The fully connected layer

In the partially connected layers, output values are not linked directly to every input pixel. In the fully connected layer, every output node connects to every node in the previous layer. This layer does the classification, using the features extracted by the earlier layers and their different filters. Convolution and pooling layers tend to use ReLU, whereas the fully connected layer usually uses softmax, which gives a probability between 0 and 1 for each class.

Invariance and Equivariance

Invariance

A function is invariant if its output stays unchanged when the input is perturbed. For images this takes a few forms:

• Translation invariance: ignoring shifts in where the target sits in the image.
• Rotation invariance: the predicted class does not change when the input is tilted.
• Robustness to noise: performance stays high on corrupted inputs.

A cat is still a cat in the top or bottom half of the image, rotated, flipped or noisy. CNNs have built-in translation invariance, which is why they suit images better than ordinary networks. It comes from combining the two layer types: the convolution reduces the image to a set of features and their positions, then max pooling lowers the resolution by keeping only the maximum of each grid and discarding where in the grid it was.

The downside of max pooling

• The network does not learn the spatial relationship between features, so it can give a false positive when all the features are present but in the wrong arrangement.
• Pooling is applied to each filter separately, so much of the relative position between features is lost. The next layer only knows which features are present.
• The effect compounds: deeper layers keep less and less spatial information.
• Data augmentation is a common remedy. It is a form of regularisation that distorts the training data (shifts, rotations, flips) so the model learns to be robust to those distortions.

Equivariance

Equivariance is often confused with invariance. An equivariant function responds to a change in the input with a matching change in the output. This is what you want in segmentation: if you rotate the image, the segmentation map should rotate with it.

Capsule networks

Pooling throws away a lot of information and gives invariance rather than equivariance. CNNs therefore struggle to relate features to each other. The "Picasso problem" shows this: a face with the mouth and eye swapped is still classified as a face. Capsule networks address this and can also be used in the deeper layers of a CNN. A capsule is a set of neurons that encodes both the orientation of a feature and its probability. They generally need much less training data, but their training procedure differs from standard backpropagation and takes much longer.

Harmonic networks

Harmonic networks build patch-wise 360-degree rotational equivariance into the representation by constraining filters to the family of circular harmonics. These are steerable filters, so every rotated version of a filter can be written as a finite linear combination of steering bases. This avoids learning many rotated copies of the same filter, guarantees rotational equivariance, and gives feature maps that transform predictably when the input rotates.

Image Classification — Cats vs Dogs

A binary CNN classifier trained on 64×64 RGB images. Achieves strong accuracy on the Cats vs. Dogs dataset using data augmentation and early stopping.

Build the Model

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import (Conv2D, MaxPool2D, BatchNormalization,
                                     Flatten, Dense, Dropout)

def build_cnn(input_shape=(64, 64, 3)):
    model = Sequential([
        Conv2D(32, (3,3), activation='relu', input_shape=input_shape),
        BatchNormalization(),
        MaxPool2D(2, 2),

        Conv2D(64, (3,3), activation='relu'),
        BatchNormalization(),
        MaxPool2D(2, 2),

        Conv2D(128, (3,3), activation='relu'),
        MaxPool2D(2, 2),

        Flatten(),
        Dense(256, activation='relu'),
        Dropout(0.5),
        Dense(1, activation='sigmoid')   # binary output
    ])
    model.compile(optimizer='adam',
                  loss='binary_crossentropy',
                  metrics=['accuracy'])
    return model

model = build_cnn()
model.summary()

Train with Data Augmentation

from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.callbacks import EarlyStopping

datagen = ImageDataGenerator(
    rescale=1./255,
    rotation_range=20,
    horizontal_flip=True,
    zoom_range=0.2,
    validation_split=0.2
)

train_gen = datagen.flow_from_directory(
    'data/', target_size=(64,64), batch_size=32,
    class_mode='binary', subset='training'
)
val_gen = datagen.flow_from_directory(
    'data/', target_size=(64,64), batch_size=32,
    class_mode='binary', subset='validation'
)

early_stop = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

history = model.fit(train_gen, validation_data=val_gen,
                    epochs=30, callbacks=[early_stop])

Live Demo — Colour Classifier

A CNN trained on 64×64 colour swatches classifies any RGB colour into one of 10 categories. Pick a colour below — the model fills a virtual 64×64 image with that colour, runs it through the CNN entirely in your browser, and returns the predicted class.

Model trained with python scripts/export_colour_cnn.py — see 5 Computer Vision/Colour_CNN.ipynb for the full training walkthrough.